2023-10-13T23:43:16.819 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-13T23:43:16.820 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --archive /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257 --name dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi --verbose --owner scheduled_dis@teuthology -- /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257/orig.config.yaml 2023-10-13T23:43:16.848 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-13T23:43:16.925 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257 branch: wip-dis-testing description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: idryomov@gmail.com first_in_suite: false job_id: '7427257' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: rbd default features: 125 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-qa sha1: 735f68d7bf91d643813a8420fc2b11441f4bd8c4 owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2449 sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f sleep_before_teardown: 0 subset: 49/99 suite: rbd suite_branch: wip-dis-testing-qa suite_path: /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa suite_relpath: qa suite_repo: https://github.com/idryomov/ceph suite_sha1: 735f68d7bf91d643813a8420fc2b11441f4bd8c4 targets: smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIbphirixlHwnOwOOVhbCQiR2axA3wmHarKaDteDBvLWHZeDd90LRhhgzWrE6nKf214ifB1/ugwsfdEyiLumX0= tasks: - install: null - ceph: null - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-13_23:35:38 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-13T23:43:16.925 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa; will attempt to use it 2023-10-13T23:43:16.926 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks 2023-10-13T23:43:16.926 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-13T23:43:16.928 INFO:teuthology.task.internal:Checking packages... 2023-10-13T23:43:16.951 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '01fff6a72a328459c1d153e5dc1de6a34e48a82f' 2023-10-13T23:43:16.952 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-13T23:43:16.952 INFO:teuthology.packaging:ref: None 2023-10-13T23:43:16.952 INFO:teuthology.packaging:tag: None 2023-10-13T23:43:16.953 INFO:teuthology.packaging:branch: wip-dis-testing 2023-10-13T23:43:16.953 INFO:teuthology.packaging:sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-13T23:43:16.953 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2023-10-13T23:43:17.178 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:43:17.178 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-13T23:43:17.186 INFO:teuthology.task.internal:no buildpackages task found 2023-10-13T23:43:17.186 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-13T23:43:17.192 INFO:teuthology.task.internal:Saving configuration 2023-10-13T23:43:17.203 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-13T23:43:17.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-13T23:43:17.232 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-13 23:37:44.811141', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIItO0uq3LWk3JssXJ+6nLiLYBjihOfbkWywGd3NMKTid'} 2023-10-13T23:43:17.232 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-13T23:43:17.238 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-13T23:43:17.239 INFO:teuthology.run_tasks:Running task console_log... 2023-10-13T23:43:17.276 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcbf902c940>, signals=[15]) 2023-10-13T23:43:17.277 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-13T23:43:17.284 INFO:teuthology.task.internal:Opening connections... 2023-10-13T23:43:17.284 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2023-10-13T23:43:17.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:43:17.365 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-13T23:43:17.372 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2023-10-13T23:43:17.458 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2023-10-13T23:43:17.459 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2023-10-13T23:43:17.502 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-13T23:43:17.503 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Ubuntu" 2023-10-13T23:43:17.503 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="22.04" 2023-10-13T23:43:17.503 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-13T23:43:17.504 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_CODENAME=jammy 2023-10-13T23:43:17.504 INFO:teuthology.orchestra.run.smithi138.stdout:ID=ubuntu 2023-10-13T23:43:17.504 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE=debian 2023-10-13T23:43:17.504 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-13T23:43:17.504 INFO:teuthology.orchestra.run.smithi138.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-13T23:43:17.505 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-13T23:43:17.505 INFO:teuthology.orchestra.run.smithi138.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-13T23:43:17.505 INFO:teuthology.orchestra.run.smithi138.stdout:UBUNTU_CODENAME=jammy 2023-10-13T23:43:17.506 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2023-10-13T23:43:17.530 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-13T23:43:17.540 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-13T23:43:17.546 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-13T23:43:17.546 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2023-10-13T23:43:17.551 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-13T23:43:17.557 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-13T23:43:17.557 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2023-10-13T23:43:17.601 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-13T23:43:17.634 INFO:teuthology.run_tasks:Running task kernel... 2023-10-13T23:43:17.647 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-13T23:43:17.648 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-13T23:43:17.649 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-13T23:43:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-13T23:43:17.655 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2023-10-13T23:43:17.703 INFO:teuthology.orchestra.run.smithi138.stdout:5.15.0-56-generic 2023-10-13T23:43:17.704 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.15.0-56-generic 2023-10-13T23:43:17.704 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2023-10-13T23:43:17.859 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2023-10-13T23:43:17.995 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-13T23:43:18.106 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:43:18.188 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-13T23:43:18.230 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [346 kB] 2023-10-13T23:43:18.301 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [856 kB] 2023-10-13T23:43:18.337 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [175 kB] 2023-10-13T23:43:18.340 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-13T23:43:18.340 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [953 kB] 2023-10-13T23:43:18.356 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-13T23:43:18.356 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [154 kB] 2023-10-13T23:43:18.359 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B] 2023-10-13T23:43:18.360 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [560 kB] 2023-10-13T23:43:18.370 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [789 kB] 2023-10-13T23:43:18.381 INFO:teuthology.orchestra.run.smithi138.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-13T23:43:18.383 INFO:teuthology.orchestra.run.smithi138.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-10-13T23:43:18.410 INFO:teuthology.orchestra.run.smithi138.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-13T23:43:18.411 INFO:teuthology.orchestra.run.smithi138.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-13T23:43:18.411 INFO:teuthology.orchestra.run.smithi138.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-13T23:43:18.411 INFO:teuthology.orchestra.run.smithi138.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-13T23:43:18.537 INFO:teuthology.orchestra.run.smithi138.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-13T23:43:18.674 INFO:teuthology.orchestra.run.smithi138.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [506 kB] 2023-10-13T23:43:18.856 INFO:teuthology.orchestra.run.smithi138.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,062 kB] 2023-10-13T23:43:18.981 INFO:teuthology.orchestra.run.smithi138.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [234 kB] 2023-10-13T23:43:18.995 INFO:teuthology.orchestra.run.smithi138.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-13T23:43:18.998 INFO:teuthology.orchestra.run.smithi138.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [974 kB] 2023-10-13T23:43:19.050 INFO:teuthology.orchestra.run.smithi138.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.3 kB] 2023-10-13T23:43:19.051 INFO:teuthology.orchestra.run.smithi138.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [157 kB] 2023-10-13T23:43:19.057 INFO:teuthology.orchestra.run.smithi138.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B] 2023-10-13T23:43:19.222 INFO:teuthology.orchestra.run.smithi138.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [658 kB] 2023-10-13T23:43:19.792 INFO:teuthology.orchestra.run.smithi138.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [991 kB] 2023-10-13T23:43:19.901 INFO:teuthology.orchestra.run.smithi138.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [217 kB] 2023-10-13T23:43:19.920 INFO:teuthology.orchestra.run.smithi138.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-13T23:43:19.922 INFO:teuthology.orchestra.run.smithi138.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-13T23:43:19.926 INFO:teuthology.orchestra.run.smithi138.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-13T23:43:19.927 INFO:teuthology.orchestra.run.smithi138.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-13T23:43:19.927 INFO:teuthology.orchestra.run.smithi138.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-13T23:43:19.928 INFO:teuthology.orchestra.run.smithi138.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-10-13T23:43:19.930 INFO:teuthology.orchestra.run.smithi138.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-10-13T23:43:19.933 INFO:teuthology.orchestra.run.smithi138.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-13T23:43:20.098 INFO:teuthology.orchestra.run.smithi138.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-13T23:43:20.180 INFO:teuthology.orchestra.run.smithi138.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-10-13T23:43:20.185 INFO:teuthology.orchestra.run.smithi138.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-10-13T23:43:20.304 INFO:teuthology.orchestra.run.smithi138.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-13T23:43:20.356 INFO:teuthology.orchestra.run.smithi138.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-10-13T23:43:24.573 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 9,489 kB in 2s (3,897 kB/s) 2023-10-13T23:43:26.069 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:43:26.087 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-13T23:43:26.149 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:43:26.333 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2023-10-13T23:43:26.334 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2023-10-13T23:43:26.488 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:43:26.488 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:43:26.489 INFO:teuthology.orchestra.run.smithi138.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:43:26.489 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:43:26.490 INFO:teuthology.orchestra.run.smithi138.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-13T23:43:26.490 INFO:teuthology.orchestra.run.smithi138.stdout: python2.7-minimal 2023-10-13T23:43:26.490 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:43:26.491 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2023-10-13T23:43:26.491 INFO:teuthology.orchestra.run.smithi138.stdout: linux-generic linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-13T23:43:26.491 INFO:teuthology.orchestra.run.smithi138.stdout: linux-headers-generic linux-image-5.15.0-86-generic 2023-10-13T23:43:26.491 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-5.15.0-86-generic linux-modules-extra-5.15.0-86-generic 2023-10-13T23:43:26.492 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2023-10-13T23:43:26.492 INFO:teuthology.orchestra.run.smithi138.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-13T23:43:26.492 INFO:teuthology.orchestra.run.smithi138.stdout:Recommended packages: 2023-10-13T23:43:26.492 INFO:teuthology.orchestra.run.smithi138.stdout: thermald 2023-10-13T23:43:26.541 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2023-10-13T23:43:26.542 INFO:teuthology.orchestra.run.smithi138.stdout: linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-13T23:43:26.542 INFO:teuthology.orchestra.run.smithi138.stdout: linux-image-5.15.0-86-generic linux-modules-5.15.0-86-generic 2023-10-13T23:43:26.542 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-extra-5.15.0-86-generic 2023-10-13T23:43:26.543 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages will be upgraded: 2023-10-13T23:43:26.543 INFO:teuthology.orchestra.run.smithi138.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-13T23:43:26.721 INFO:teuthology.orchestra.run.smithi138.stdout:3 upgraded, 5 newly installed, 0 to remove and 268 not upgraded. 2023-10-13T23:43:26.721 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 113 MB of archives. 2023-10-13T23:43:26.722 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-13T23:43:26.722 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-86-generic amd64 5.15.0-86.96 [22.4 MB] 2023-10-13T23:43:28.107 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-86-generic amd64 5.15.0-86.96 [11.5 MB] 2023-10-13T23:43:28.452 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-86-generic amd64 5.15.0-86.96 [63.9 MB] 2023-10-13T23:43:30.652 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.86.83 [1,702 B] 2023-10-13T23:43:30.652 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.86.83 [2,472 B] 2023-10-13T23:43:30.653 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86 all 5.15.0-86.96 [12.3 MB] 2023-10-13T23:43:31.886 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86-generic amd64 5.15.0-86.96 [2,882 kB] 2023-10-13T23:43:32.009 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.86.83 [2,328 B] 2023-10-13T23:43:32.392 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 113 MB in 5s (20.7 MB/s) 2023-10-13T23:43:32.579 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-5.15.0-86-generic. 2023-10-13T23:43:34.607 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-13T23:43:34.612 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../0-linux-modules-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:43:34.693 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:43:37.227 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-image-5.15.0-86-generic. 2023-10-13T23:43:37.248 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../1-linux-image-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:43:37.334 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:43:37.805 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-86-generic. 2023-10-13T23:43:37.826 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:43:37.863 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:43:44.513 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../3-linux-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:43:44.632 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:43:44.957 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:43:45.077 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-image-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:43:45.397 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-headers-5.15.0-86. 2023-10-13T23:43:45.419 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../5-linux-headers-5.15.0-86_5.15.0-86.96_all.deb ... 2023-10-13T23:43:45.463 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-13T23:43:51.494 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package linux-headers-5.15.0-86-generic. 2023-10-13T23:43:51.531 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../6-linux-headers-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:43:51.578 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:43:54.649 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:43:54.775 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking linux-headers-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:43:55.124 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-13T23:43:55.230 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:43:55.331 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-headers-generic (5.15.0.86.83) ... 2023-10-13T23:43:55.433 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:43:56.770 INFO:teuthology.orchestra.run.smithi138.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:43:56.771 INFO:teuthology.orchestra.run.smithi138.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-13T23:43:56.771 INFO:teuthology.orchestra.run.smithi138.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:43:56.771 INFO:teuthology.orchestra.run.smithi138.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-86-generic 2023-10-13T23:43:56.930 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:43:58.306 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:44:00.008 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-image-generic (5.15.0.86.83) ... 2023-10-13T23:44:00.109 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up linux-generic (5.15.0.86.83) ... 2023-10-13T23:44:00.210 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:44:00.268 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-13T23:44:00.269 INFO:teuthology.orchestra.run.smithi138.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:44:14.685 INFO:teuthology.orchestra.run.smithi138.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-13T23:44:14.685 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub' 2023-10-13T23:44:14.697 INFO:teuthology.orchestra.run.smithi138.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-13T23:44:14.723 INFO:teuthology.orchestra.run.smithi138.stdout:Generating grub configuration file ... 2023-10-13T23:44:15.093 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:44:15.113 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:44:15.462 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:44:15.467 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-13T23:44:15.531 INFO:teuthology.orchestra.run.smithi138.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-13T23:44:15.537 INFO:teuthology.orchestra.run.smithi138.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-13T23:44:15.686 INFO:teuthology.orchestra.run.smithi138.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-13T23:44:15.686 INFO:teuthology.orchestra.run.smithi138.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-13T23:44:15.687 INFO:teuthology.orchestra.run.smithi138.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-13T23:44:15.726 INFO:teuthology.orchestra.run.smithi138.stdout:done 2023-10-13T23:44:16.152 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic 2023-10-13T23:44:16.179 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic 2023-10-13T23:44:16.180 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2023-10-13T23:44:16.180 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2023-10-13T23:44:16.180 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2023-10-13T23:44:16.180 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 20 2023-10-13T23:44:16.181 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2023-10-13T23:44:16.181 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2023-10-13T23:44:16.181 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta 2023-10-13T23:44:16.181 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.86.83 2023-10-13T23:44:16.181 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-13T23:44:16.182 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-13T23:44:16.182 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2023-10-13T23:44:16.182 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2023-10-13T23:44:16.182 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2023-10-13T23:44:16.182 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2023-10-13T23:44:16.183 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-86-generic 2023-10-13T23:44:16.183 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-13T23:44:16.183 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-13T23:44:16.183 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-13T23:44:16.184 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2023-10-13T23:44:16.184 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get clean 2023-10-13T23:44:16.368 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2023-10-13T23:44:16.493 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:44:16.494 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-13T23:44:16.501 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-13T23:44:16.518 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-13T23:44:17.772 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:44:17.790 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-13T23:44:17.839 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:44:18.009 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2023-10-13T23:44:18.010 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2023-10-13T23:44:18.153 INFO:teuthology.orchestra.run.smithi138.stdout:linux-image-generic is already the newest version (5.15.0.86.83). 2023-10-13T23:44:18.153 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:44:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:44:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:44:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:44:18.154 INFO:teuthology.orchestra.run.smithi138.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-13T23:44:18.155 INFO:teuthology.orchestra.run.smithi138.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-13T23:44:18.155 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:44:18.189 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 0 newly installed, 0 to remove and 268 not upgraded. 2023-10-13T23:44:18.191 DEBUG:teuthology.orchestra.run.smithi138:> dpkg -s linux-image-generic 2023-10-13T23:44:18.249 INFO:teuthology.orchestra.run.smithi138.stdout:Package: linux-image-generic 2023-10-13T23:44:18.250 INFO:teuthology.orchestra.run.smithi138.stdout:Status: install ok installed 2023-10-13T23:44:18.250 INFO:teuthology.orchestra.run.smithi138.stdout:Priority: optional 2023-10-13T23:44:18.250 INFO:teuthology.orchestra.run.smithi138.stdout:Section: kernel 2023-10-13T23:44:18.250 INFO:teuthology.orchestra.run.smithi138.stdout:Installed-Size: 20 2023-10-13T23:44:18.250 INFO:teuthology.orchestra.run.smithi138.stdout:Maintainer: Ubuntu Kernel Team 2023-10-13T23:44:18.251 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: amd64 2023-10-13T23:44:18.251 INFO:teuthology.orchestra.run.smithi138.stdout:Source: linux-meta 2023-10-13T23:44:18.251 INFO:teuthology.orchestra.run.smithi138.stdout:Version: 5.15.0.86.83 2023-10-13T23:44:18.251 INFO:teuthology.orchestra.run.smithi138.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-13T23:44:18.251 INFO:teuthology.orchestra.run.smithi138.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-13T23:44:18.252 INFO:teuthology.orchestra.run.smithi138.stdout:Recommends: thermald 2023-10-13T23:44:18.252 INFO:teuthology.orchestra.run.smithi138.stdout:Description: Generic Linux kernel image 2023-10-13T23:44:18.252 INFO:teuthology.orchestra.run.smithi138.stdout: This package will always depend on the latest generic kernel image 2023-10-13T23:44:18.252 INFO:teuthology.orchestra.run.smithi138.stdout: available. 2023-10-13T23:44:18.253 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2023-10-13T23:44:18.294 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.60qux5tdxa 2023-10-13T23:44:18.295 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub/grub.cfg /tmp/tmp.60qux5tdxa 2023-10-13T23:44:18.350 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.60qux5tdxa 2023-10-13T23:44:18.458 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.60qux5tdxa is 10KB 2023-10-13T23:44:18.471 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.60qux5tdxa 2023-10-13T23:44:18.479 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-13T23:44:18.533 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:44:18.534 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-13T23:44:18.534 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-13T23:44:18.598 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-86-generic 2023-10-13T23:44:18.599 DEBUG:teuthology.orchestra.run.smithi138:> sudo update-grub 2023-10-13T23:44:19.874 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub' 2023-10-13T23:44:19.876 INFO:teuthology.orchestra.run.smithi138.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-13T23:44:19.883 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2023-10-13T23:44:20.076 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:44:20.082 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:44:20.391 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:44:20.397 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-13T23:44:20.441 INFO:teuthology.orchestra.run.smithi138.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-13T23:44:20.446 INFO:teuthology.orchestra.run.smithi138.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-13T23:44:20.569 INFO:teuthology.orchestra.run.smithi138.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-13T23:44:20.569 INFO:teuthology.orchestra.run.smithi138.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-13T23:44:20.569 INFO:teuthology.orchestra.run.smithi138.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-13T23:44:20.580 INFO:teuthology.orchestra.run.smithi138.stderr:done 2023-10-13T23:44:20.591 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2023-10-13T23:44:50.622 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-13T23:44:50.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2023-10-13T23:44:50.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:45:50.686 DEBUG:teuthology.orchestra.remote:timed out 2023-10-13T23:45:59.694 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2023-10-13T23:45:59.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:45:59.697 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2023-10-13T23:46:11.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2023-10-13T23:46:11.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:46:11.918 DEBUG:teuthology.orchestra.run.smithi138:> true 2023-10-13T23:46:12.576 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2023-10-13T23:46:12.576 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-86-generic"... 2023-10-13T23:46:12.576 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2023-10-13T23:46:12.623 INFO:teuthology.orchestra.run.smithi138.stdout:5.15.0-86-generic 2023-10-13T23:46:12.624 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-86-generic vs 5.15.0-86-generic 2023-10-13T23:46:12.624 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-13T23:46:12.624 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-13T23:46:13.626 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-13T23:46:13.626 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-13T23:46:13.963 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2023-10-13T23:46:13.982 DEBUG:teuthology.parallel:result is None 2023-10-13T23:46:13.982 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-13T23:46:13.992 INFO:teuthology.task.internal:Creating test directory... 2023-10-13T23:46:13.993 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-13T23:46:13.999 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-13T23:46:14.010 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-13T23:46:14.020 INFO:teuthology.task.internal:Creating archive directory... 2023-10-13T23:46:14.020 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-13T23:46:14.103 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-13T23:46:14.343 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-13T23:46:14.343 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-13T23:46:14.379 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-13T23:46:14.388 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-13T23:46:14.390 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-13T23:46:14.398 INFO:teuthology.task.internal:Configuring sudo... 2023-10-13T23:46:14.399 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-13T23:46:14.450 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-13T23:46:14.460 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-13T23:46:14.460 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-13T23:46:14.496 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-13T23:46:14.544 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-13T23:46:14.592 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:46:14.593 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-13T23:46:14.659 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2023-10-13T23:46:14.798 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-13T23:46:14.806 INFO:teuthology.task.internal:Starting timer... 2023-10-13T23:46:14.806 INFO:teuthology.run_tasks:Running task pcp... 2023-10-13T23:46:14.818 INFO:teuthology.run_tasks:Running task selinux... 2023-10-13T23:46:14.827 DEBUG:teuthology.task.selinux:Excluding smithi138: OS 'ubuntu' does not support SELinux 2023-10-13T23:46:14.828 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-13T23:46:14.828 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-13T23:46:14.828 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-13T23:46:14.829 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-13T23:46:14.839 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-13T23:46:14.839 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-13T23:46:14.858 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-13T23:46:14.859 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-13T23:53:55.554 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2023-10-13T23:53:55.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2023-10-13T23:53:55.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:53:55.634 DEBUG:teuthology.orchestra.run.smithi138:> true 2023-10-13T23:53:55.734 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2023-10-13T23:53:55.734 INFO:teuthology.run_tasks:Running task clock... 2023-10-13T23:53:55.745 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-13T23:53:55.745 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-13T23:53:55.746 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-13T23:53:55.802 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-13T23:53:55.802 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Command line: ntpd -gq 2023-10-13T23:53:55.803 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: ---------------------------------------------------- 2023-10-13T23:53:55.803 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: ntp-4 is maintained by Network Time Foundation, 2023-10-13T23:53:55.803 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-13T23:53:55.803 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: corporation. Support and training for ntp-4 are 2023-10-13T23:53:55.803 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: available at https://www.nwtime.org/support 2023-10-13T23:53:55.804 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: ---------------------------------------------------- 2023-10-13T23:53:55.804 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: proto: precision = 0.069 usec (-24) 2023-10-13T23:53:55.804 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: basedate set to 2022-02-04 2023-10-13T23:53:55.805 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: gps base set to 2022-02-06 (week 2196) 2023-10-13T23:53:55.805 INFO:teuthology.orchestra.run.smithi138.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-13T23:53:55.806 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-13T23:53:55.806 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: restrict ::: KOD does nothing without LIMITED. 2023-10-13T23:53:55.806 INFO:teuthology.orchestra.run.smithi138.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-13T23:53:55.807 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Listen and drop on 0 v6wildcard [::]:123 2023-10-13T23:53:55.808 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-13T23:53:55.808 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Listen normally on 2 lo 127.0.0.1:123 2023-10-13T23:53:55.808 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Listen normally on 3 enp3s0f1 172.21.15.138:123 2023-10-13T23:53:55.808 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Listen normally on 4 lo [::1]:123 2023-10-13T23:53:55.808 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f71%5]:123 2023-10-13T23:53:55.809 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:53:55 ntpd[27978]: Listening on routing socket on fd #22 for interface updates 2023-10-13T23:53:56.802 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:56 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:56.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:56 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:56.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:56 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:56.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:56 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:56.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:56 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:53:56.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:56 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:53:58.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:58.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:58.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:58.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:58.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:58.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:58.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:53:58.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:53:58.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:58.805 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:58.805 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:53:58.805 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:58 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:53:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:59.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:53:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:53:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:53:59.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:53:59 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:54:00.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:00.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:00.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:00.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:00.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:00.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:00.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:54:00.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:54:00.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:00.805 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:00.805 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:54:00.805 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:00 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:54:01.802 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:01.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:01 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:01.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:01.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:01 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:01.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:01 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:54:01.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:01 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:54:02.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:02.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:02.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:02.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:02.803 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:02.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:02.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:54:02.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:54:02.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-13T23:54:02.804 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: can't open /var/log/ntpstats/rawstats.20231013: Permission denied 2023-10-13T23:54:02.805 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-13T23:54:02.805 INFO:teuthology.orchestra.run.smithi138.stdout:13 Oct 23:54:02 ntpd[27978]: ntpd: time slew -0.000830 s 2023-10-13T23:54:02.805 INFO:teuthology.orchestra.run.smithi138.stdout:ntpd: time slew -0.000830s 2023-10-13T23:54:02.806 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: can't open /var/log/ntpstats/peerstats.20231013: Permission denied 2023-10-13T23:54:02.806 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-13T23:54:02.806 INFO:teuthology.orchestra.run.smithi138.stderr:13 Oct 23:54:02 ntpd[27978]: can't open /var/log/ntpstats/loopstats.20231013: Permission denied 2023-10-13T23:54:02.891 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2023-10-13T23:54:02.891 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2023-10-13T23:54:02.891 INFO:teuthology.orchestra.run.smithi138.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-13T23:54:02.891 INFO:teuthology.orchestra.run.smithi138.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-13T23:54:02.892 INFO:teuthology.orchestra.run.smithi138.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-13T23:54:02.892 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-13T23:54:02.893 INFO:teuthology.run_tasks:Running task install... 2023-10-13T23:54:02.901 DEBUG:teuthology.task.install:project ceph 2023-10-13T23:54:02.902 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}} 2023-10-13T23:54:02.902 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'} 2023-10-13T23:54:02.902 INFO:teuthology.task.install:Using flavor: default 2023-10-13T23:54:02.908 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-13T23:54:02.908 INFO:teuthology.task.install:extra packages: [] 2023-10-13T23:54:02.909 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-key list | grep Ceph 2023-10-13T23:54:02.990 INFO:teuthology.orchestra.run.smithi138.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-13T23:54:03.016 INFO:teuthology.orchestra.run.smithi138.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-13T23:54:03.017 INFO:teuthology.orchestra.run.smithi138.stdout:uid [ unknown] Ceph.com (release key) 2023-10-13T23:54:03.017 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-13T23:54:03.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-13T23:54:03.162 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default/ 2023-10-13T23:54:03.163 INFO:teuthology.task.install.deb:Package version is 18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:54:03.429 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:54:03.429 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-13T23:54:03.444 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2023-10-13T23:54:03.604 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-13T23:54:03.728 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:54:03.814 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-13T23:54:03.815 INFO:teuthology.orchestra.run.smithi138.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy InRelease 2023-10-13T23:54:03.870 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-13T23:54:03.892 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-13T23:54:03.934 INFO:teuthology.orchestra.run.smithi138.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-13T23:54:03.998 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2023-10-13T23:54:04.068 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-10-13T23:54:04.149 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 28.5 kB in 1s (54.0 kB/s) 2023-10-13T23:54:05.041 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:54:05.057 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6724-g01fff6a7-1jammy cephadm=18.0.0-6724-g01fff6a7-1jammy ceph-mds=18.0.0-6724-g01fff6a7-1jammy ceph-mgr=18.0.0-6724-g01fff6a7-1jammy ceph-common=18.0.0-6724-g01fff6a7-1jammy ceph-fuse=18.0.0-6724-g01fff6a7-1jammy ceph-test=18.0.0-6724-g01fff6a7-1jammy ceph-volume=18.0.0-6724-g01fff6a7-1jammy radosgw=18.0.0-6724-g01fff6a7-1jammy python3-rados=18.0.0-6724-g01fff6a7-1jammy python3-rgw=18.0.0-6724-g01fff6a7-1jammy python3-cephfs=18.0.0-6724-g01fff6a7-1jammy python3-rbd=18.0.0-6724-g01fff6a7-1jammy libcephfs2=18.0.0-6724-g01fff6a7-1jammy libcephfs-dev=18.0.0-6724-g01fff6a7-1jammy librados2=18.0.0-6724-g01fff6a7-1jammy librbd1=18.0.0-6724-g01fff6a7-1jammy rbd-fuse=18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:54:05.119 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:54:05.309 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2023-10-13T23:54:05.310 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2023-10-13T23:54:05.478 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:54:05.479 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:54:05.479 INFO:teuthology.orchestra.run.smithi138.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:54:05.479 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:54:05.479 INFO:teuthology.orchestra.run.smithi138.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-13T23:54:05.480 INFO:teuthology.orchestra.run.smithi138.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-13T23:54:05.481 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:54:05.481 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2023-10-13T23:54:05.481 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-13T23:54:05.481 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-13T23:54:05.481 INFO:teuthology.orchestra.run.smithi138.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-13T23:54:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-13T23:54:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-13T23:54:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-13T23:54:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-13T23:54:05.482 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-13T23:54:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-13T23:54:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-13T23:54:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-13T23:54:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-13T23:54:05.483 INFO:teuthology.orchestra.run.smithi138.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-13T23:54:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-13T23:54:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-13T23:54:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-13T23:54:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-13T23:54:05.484 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-13T23:54:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-13T23:54:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-13T23:54:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-13T23:54:05.485 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-13T23:54:05.486 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2023-10-13T23:54:05.486 INFO:teuthology.orchestra.run.smithi138.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-13T23:54:05.486 INFO:teuthology.orchestra.run.smithi138.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-13T23:54:05.486 INFO:teuthology.orchestra.run.smithi138.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-13T23:54:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-13T23:54:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-13T23:54:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-13T23:54:05.487 INFO:teuthology.orchestra.run.smithi138.stdout: python3-watchdog 2023-10-13T23:54:05.487 INFO:teuthology.orchestra.run.smithi138.stdout:Recommended packages: 2023-10-13T23:54:05.488 INFO:teuthology.orchestra.run.smithi138.stdout: btrfs-tools 2023-10-13T23:54:05.580 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2023-10-13T23:54:05.580 INFO:teuthology.orchestra.run.smithi138.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-13T23:54:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-13T23:54:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-13T23:54:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-13T23:54:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-13T23:54:05.581 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-13T23:54:05.582 INFO:teuthology.orchestra.run.smithi138.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-13T23:54:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-13T23:54:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-13T23:54:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-13T23:54:05.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-13T23:54:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-13T23:54:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-13T23:54:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-13T23:54:05.584 INFO:teuthology.orchestra.run.smithi138.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-13T23:54:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-13T23:54:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-13T23:54:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-13T23:54:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-13T23:54:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-13T23:54:05.585 INFO:teuthology.orchestra.run.smithi138.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-13T23:54:05.586 INFO:teuthology.orchestra.run.smithi138.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-13T23:54:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-13T23:54:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-13T23:54:05.587 INFO:teuthology.orchestra.run.smithi138.stdout: socat xmlstarlet 2023-10-13T23:54:05.644 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 109 newly installed, 0 to remove and 256 not upgraded. 2023-10-13T23:54:05.644 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 184 MB of archives. 2023-10-13T23:54:05.644 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-13T23:54:05.645 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-13T23:54:05.710 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-13T23:54:05.752 INFO:teuthology.orchestra.run.smithi138.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-13T23:54:05.754 INFO:teuthology.orchestra.run.smithi138.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-13T23:54:05.772 INFO:teuthology.orchestra.run.smithi138.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-13T23:54:05.828 INFO:teuthology.orchestra.run.smithi138.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-13T23:54:05.834 INFO:teuthology.orchestra.run.smithi138.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-13T23:54:05.846 INFO:teuthology.orchestra.run.smithi138.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-13T23:54:05.850 INFO:teuthology.orchestra.run.smithi138.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-13T23:54:05.850 INFO:teuthology.orchestra.run.smithi138.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-13T23:54:05.851 INFO:teuthology.orchestra.run.smithi138.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-13T23:54:05.859 INFO:teuthology.orchestra.run.smithi138.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-13T23:54:05.860 INFO:teuthology.orchestra.run.smithi138.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-13T23:54:05.860 INFO:teuthology.orchestra.run.smithi138.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-13T23:54:05.861 INFO:teuthology.orchestra.run.smithi138.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-13T23:54:05.872 INFO:teuthology.orchestra.run.smithi138.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-13T23:54:05.873 INFO:teuthology.orchestra.run.smithi138.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-13T23:54:05.873 INFO:teuthology.orchestra.run.smithi138.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-13T23:54:05.892 INFO:teuthology.orchestra.run.smithi138.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-13T23:54:05.892 INFO:teuthology.orchestra.run.smithi138.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-13T23:54:05.892 INFO:teuthology.orchestra.run.smithi138.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-13T23:54:05.896 INFO:teuthology.orchestra.run.smithi138.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-13T23:54:05.896 INFO:teuthology.orchestra.run.smithi138.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-13T23:54:05.897 INFO:teuthology.orchestra.run.smithi138.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-13T23:54:05.898 INFO:teuthology.orchestra.run.smithi138.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-13T23:54:05.911 INFO:teuthology.orchestra.run.smithi138.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-13T23:54:05.912 INFO:teuthology.orchestra.run.smithi138.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-13T23:54:05.912 INFO:teuthology.orchestra.run.smithi138.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-13T23:54:05.931 INFO:teuthology.orchestra.run.smithi138.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-13T23:54:05.932 INFO:teuthology.orchestra.run.smithi138.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-13T23:54:05.932 INFO:teuthology.orchestra.run.smithi138.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-13T23:54:05.940 INFO:teuthology.orchestra.run.smithi138.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-13T23:54:05.940 INFO:teuthology.orchestra.run.smithi138.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-13T23:54:05.941 INFO:teuthology.orchestra.run.smithi138.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-13T23:54:05.942 INFO:teuthology.orchestra.run.smithi138.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-13T23:54:05.950 INFO:teuthology.orchestra.run.smithi138.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-13T23:54:05.953 INFO:teuthology.orchestra.run.smithi138.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-13T23:54:05.954 INFO:teuthology.orchestra.run.smithi138.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-13T23:54:05.970 INFO:teuthology.orchestra.run.smithi138.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-13T23:54:05.971 INFO:teuthology.orchestra.run.smithi138.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-13T23:54:05.974 INFO:teuthology.orchestra.run.smithi138.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-13T23:54:05.975 INFO:teuthology.orchestra.run.smithi138.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-13T23:54:05.978 INFO:teuthology.orchestra.run.smithi138.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-13T23:54:06.041 INFO:teuthology.orchestra.run.smithi138.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-13T23:54:06.054 INFO:teuthology.orchestra.run.smithi138.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-13T23:54:06.055 INFO:teuthology.orchestra.run.smithi138.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-13T23:54:06.056 INFO:teuthology.orchestra.run.smithi138.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-13T23:54:06.056 INFO:teuthology.orchestra.run.smithi138.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-13T23:54:06.057 INFO:teuthology.orchestra.run.smithi138.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-13T23:54:06.066 INFO:teuthology.orchestra.run.smithi138.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6724-g01fff6a7-1jammy [3,450 kB] 2023-10-13T23:54:06.077 INFO:teuthology.orchestra.run.smithi138.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-13T23:54:06.079 INFO:teuthology.orchestra.run.smithi138.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-13T23:54:06.079 INFO:teuthology.orchestra.run.smithi138.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-13T23:54:06.095 INFO:teuthology.orchestra.run.smithi138.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-13T23:54:06.097 INFO:teuthology.orchestra.run.smithi138.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-13T23:54:06.098 INFO:teuthology.orchestra.run.smithi138.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-13T23:54:06.099 INFO:teuthology.orchestra.run.smithi138.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-13T23:54:06.099 INFO:teuthology.orchestra.run.smithi138.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-13T23:54:06.100 INFO:teuthology.orchestra.run.smithi138.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-13T23:54:06.117 INFO:teuthology.orchestra.run.smithi138.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-13T23:54:06.117 INFO:teuthology.orchestra.run.smithi138.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-13T23:54:06.121 INFO:teuthology.orchestra.run.smithi138.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-13T23:54:06.123 INFO:teuthology.orchestra.run.smithi138.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-13T23:54:06.125 INFO:teuthology.orchestra.run.smithi138.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-13T23:54:06.136 INFO:teuthology.orchestra.run.smithi138.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-13T23:54:06.140 INFO:teuthology.orchestra.run.smithi138.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-13T23:54:06.143 INFO:teuthology.orchestra.run.smithi138.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-13T23:54:06.148 INFO:teuthology.orchestra.run.smithi138.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-13T23:54:06.152 INFO:teuthology.orchestra.run.smithi138.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-13T23:54:06.155 INFO:teuthology.orchestra.run.smithi138.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-13T23:54:06.156 INFO:teuthology.orchestra.run.smithi138.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-13T23:54:06.157 INFO:teuthology.orchestra.run.smithi138.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-13T23:54:06.158 INFO:teuthology.orchestra.run.smithi138.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-13T23:54:06.159 INFO:teuthology.orchestra.run.smithi138.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-13T23:54:06.175 INFO:teuthology.orchestra.run.smithi138.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-13T23:54:06.175 INFO:teuthology.orchestra.run.smithi138.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-13T23:54:06.176 INFO:teuthology.orchestra.run.smithi138.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-13T23:54:06.178 INFO:teuthology.orchestra.run.smithi138.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-13T23:54:06.179 INFO:teuthology.orchestra.run.smithi138.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-13T23:54:06.518 INFO:teuthology.orchestra.run.smithi138.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6724-g01fff6a7-1jammy [3,130 kB] 2023-10-13T23:54:06.725 INFO:teuthology.orchestra.run.smithi138.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6724-g01fff6a7-1jammy [688 kB] 2023-10-13T23:54:06.747 INFO:teuthology.orchestra.run.smithi138.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6724-g01fff6a7-1jammy [356 kB] 2023-10-13T23:54:06.778 INFO:teuthology.orchestra.run.smithi138.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6724-g01fff6a7-1jammy [32.8 kB] 2023-10-13T23:54:06.779 INFO:teuthology.orchestra.run.smithi138.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6724-g01fff6a7-1jammy [180 kB] 2023-10-13T23:54:06.783 INFO:teuthology.orchestra.run.smithi138.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6724-g01fff6a7-1jammy [62.4 kB] 2023-10-13T23:54:06.785 INFO:teuthology.orchestra.run.smithi138.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6724-g01fff6a7-1jammy [330 kB] 2023-10-13T23:54:06.792 INFO:teuthology.orchestra.run.smithi138.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6724-g01fff6a7-1jammy [6,222 kB] 2023-10-13T23:54:07.140 INFO:teuthology.orchestra.run.smithi138.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6724-g01fff6a7-1jammy [112 kB] 2023-10-13T23:54:07.143 INFO:teuthology.orchestra.run.smithi138.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6724-g01fff6a7-1jammy [439 kB] 2023-10-13T23:54:07.157 INFO:teuthology.orchestra.run.smithi138.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6724-g01fff6a7-1jammy [22.6 MB] 2023-10-13T23:54:08.002 INFO:teuthology.orchestra.run.smithi138.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6724-g01fff6a7-1jammy [6,853 kB] 2023-10-13T23:54:08.178 INFO:teuthology.orchestra.run.smithi138.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6724-g01fff6a7-1jammy [238 kB] 2023-10-13T23:54:08.184 INFO:teuthology.orchestra.run.smithi138.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6724-g01fff6a7-1jammy [128 kB] 2023-10-13T23:54:08.188 INFO:teuthology.orchestra.run.smithi138.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6724-g01fff6a7-1jammy [1,496 kB] 2023-10-13T23:54:08.239 INFO:teuthology.orchestra.run.smithi138.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6724-g01fff6a7-1jammy [9,863 kB] 2023-10-13T23:54:08.563 INFO:teuthology.orchestra.run.smithi138.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6724-g01fff6a7-1jammy [31.7 MB] 2023-10-13T23:54:09.793 INFO:teuthology.orchestra.run.smithi138.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6724-g01fff6a7-1jammy [14.0 kB] 2023-10-13T23:54:09.793 INFO:teuthology.orchestra.run.smithi138.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6724-g01fff6a7-1jammy [839 kB] 2023-10-13T23:54:09.810 INFO:teuthology.orchestra.run.smithi138.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6724-g01fff6a7-1jammy [2,146 kB] 2023-10-13T23:54:09.848 INFO:teuthology.orchestra.run.smithi138.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6724-g01fff6a7-1jammy [283 kB] 2023-10-13T23:54:09.852 INFO:teuthology.orchestra.run.smithi138.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6724-g01fff6a7-1jammy [138 kB] 2023-10-13T23:54:09.854 INFO:teuthology.orchestra.run.smithi138.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6724-g01fff6a7-1jammy [1,906 kB] 2023-10-13T23:54:09.876 INFO:teuthology.orchestra.run.smithi138.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6724-g01fff6a7-1jammy [8,625 kB] 2023-10-13T23:54:10.158 INFO:teuthology.orchestra.run.smithi138.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6724-g01fff6a7-1jammy [14.2 kB] 2023-10-13T23:54:10.158 INFO:teuthology.orchestra.run.smithi138.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6724-g01fff6a7-1jammy [47.8 MB] 2023-10-13T23:54:11.990 INFO:teuthology.orchestra.run.smithi138.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6724-g01fff6a7-1jammy [120 kB] 2023-10-13T23:54:12.068 INFO:teuthology.orchestra.run.smithi138.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6724-g01fff6a7-1jammy [38.3 kB] 2023-10-13T23:54:12.069 INFO:teuthology.orchestra.run.smithi138.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6724-g01fff6a7-1jammy [12.3 MB] 2023-10-13T23:54:12.383 INFO:teuthology.orchestra.run.smithi138.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6724-g01fff6a7-1jammy [91.4 kB] 2023-10-13T23:54:12.929 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 184 MB in 7s (27.0 MB/s) 2023-10-13T23:54:13.068 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-13T23:54:13.164 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2023-10-13T23:54:13.169 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-13T23:54:13.250 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-13T23:54:13.586 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-13T23:54:13.653 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-13T23:54:14.031 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-13T23:54:14.058 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-13T23:54:14.089 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-13T23:54:14.417 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-13T23:54:14.444 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-13T23:54:14.484 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-13T23:54:14.862 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-13T23:54:14.889 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-13T23:54:14.929 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-13T23:54:15.266 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-13T23:54:15.281 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-13T23:54:15.315 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-13T23:54:15.660 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-13T23:54:15.687 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-13T23:54:15.727 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-13T23:54:16.088 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-13T23:54:16.114 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-13T23:54:16.155 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-13T23:54:16.483 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librados2. 2023-10-13T23:54:16.509 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../008-librados2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:16.549 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librados2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:16.977 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librbd1. 2023-10-13T23:54:16.992 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../009-librbd1_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:17.027 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librbd1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:17.438 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libcephfs2. 2023-10-13T23:54:17.464 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:17.513 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libcephfs2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:17.824 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rados. 2023-10-13T23:54:17.839 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:17.882 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rados (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:18.185 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-13T23:54:18.211 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:18.251 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-ceph-argparse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:18.454 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cephfs. 2023-10-13T23:54:18.468 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:18.504 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cephfs (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:18.823 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-13T23:54:18.849 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:18.890 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:19.418 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-13T23:54:19.444 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-13T23:54:19.484 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-13T23:54:19.804 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-prettytable. 2023-10-13T23:54:19.830 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-13T23:54:19.870 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-13T23:54:20.123 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rbd. 2023-10-13T23:54:20.138 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:20.173 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rbd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:20.484 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-13T23:54:20.511 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-13T23:54:20.551 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-13T23:54:20.887 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package librgw2. 2023-10-13T23:54:20.913 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../019-librgw2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:20.953 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking librgw2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:21.514 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rgw. 2023-10-13T23:54:21.530 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:21.573 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:21.842 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-13T23:54:21.857 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-13T23:54:21.892 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-13T23:54:22.195 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libradosstriper1. 2023-10-13T23:54:22.221 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:22.261 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libradosstriper1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:22.640 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-common. 2023-10-13T23:54:22.666 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:22.706 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:24.029 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-base. 2023-10-13T23:54:24.055 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:24.105 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-base (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:25.008 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-13T23:54:25.035 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-13T23:54:25.075 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-13T23:54:25.405 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cheroot. 2023-10-13T23:54:25.429 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-13T23:54:25.469 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-13T23:54:25.806 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-13T23:54:25.832 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-13T23:54:25.872 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-13T23:54:26.125 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-13T23:54:26.151 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-13T23:54:26.183 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-13T23:54:26.494 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-13T23:54:26.521 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-13T23:54:26.561 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-13T23:54:26.872 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-tempora. 2023-10-13T23:54:26.899 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-13T23:54:26.939 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-13T23:54:27.234 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-portend. 2023-10-13T23:54:27.260 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-13T23:54:27.300 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-13T23:54:27.603 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-13T23:54:27.628 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-13T23:54:27.669 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-13T23:54:27.981 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-13T23:54:28.000 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-13T23:54:28.039 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-13T23:54:28.392 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-natsort. 2023-10-13T23:54:28.419 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-13T23:54:28.459 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-13T23:54:28.745 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-logutils. 2023-10-13T23:54:28.772 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-13T23:54:28.812 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-13T23:54:29.065 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-13T23:54:29.080 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-13T23:54:29.140 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-13T23:54:29.443 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-mako. 2023-10-13T23:54:29.469 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-13T23:54:29.509 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-13T23:54:30.037 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-13T23:54:30.065 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-13T23:54:30.104 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-13T23:54:30.415 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-13T23:54:30.442 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-13T23:54:30.482 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-13T23:54:30.743 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-webob. 2023-10-13T23:54:30.769 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-13T23:54:30.801 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-13T23:54:31.112 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-waitress. 2023-10-13T23:54:31.129 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-13T23:54:31.174 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-13T23:54:31.457 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-tempita. 2023-10-13T23:54:31.483 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-13T23:54:31.523 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-13T23:54:31.826 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-paste. 2023-10-13T23:54:31.852 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-13T23:54:31.909 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-13T23:54:32.212 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-13T23:54:32.229 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-13T23:54:32.262 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-13T23:54:32.565 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-13T23:54:32.592 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-13T23:54:32.632 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-13T23:54:32.951 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-webtest. 2023-10-13T23:54:32.978 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-13T23:54:33.018 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-13T23:54:33.330 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pecan. 2023-10-13T23:54:33.355 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-13T23:54:33.396 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-13T23:54:33.759 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-certifi. 2023-10-13T23:54:33.786 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-13T23:54:33.826 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-13T23:54:34.112 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-idna. 2023-10-13T23:54:34.138 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-13T23:54:34.170 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-13T23:54:34.465 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-urllib3. 2023-10-13T23:54:34.491 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-13T23:54:34.531 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-13T23:54:34.826 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-requests. 2023-10-13T23:54:34.853 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-13T23:54:34.893 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-13T23:54:35.187 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-13T23:54:35.213 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-13T23:54:35.270 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-13T23:54:35.557 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-13T23:54:35.572 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:35.607 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:35.970 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-13T23:54:35.997 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:36.036 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:36.364 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-openssl. 2023-10-13T23:54:36.391 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-13T23:54:36.431 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-13T23:54:36.725 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr. 2023-10-13T23:54:36.749 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:36.809 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:37.204 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mon. 2023-10-13T23:54:37.231 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:37.270 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mon (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:38.207 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-osd. 2023-10-13T23:54:38.234 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:38.274 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-osd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:40.652 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph. 2023-10-13T23:54:40.668 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../059-ceph_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:40.702 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:40.996 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-fuse. 2023-10-13T23:54:41.023 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:41.063 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:41.466 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mds. 2023-10-13T23:54:41.493 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:41.533 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mds (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:41.902 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package cephadm. 2023-10-13T23:54:41.930 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../062-cephadm_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:41.969 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:42.230 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-13T23:54:42.257 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-13T23:54:42.297 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-13T23:54:42.625 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-13T23:54:42.652 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-13T23:54:42.691 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-13T23:54:43.755 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-babel. 2023-10-13T23:54:43.782 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-13T23:54:43.821 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-13T23:54:44.141 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-jinja2. 2023-10-13T23:54:44.168 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-13T23:54:44.207 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-13T23:54:44.578 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-13T23:54:44.604 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:44.644 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:44.981 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-13T23:54:45.008 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-13T23:54:45.056 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-13T23:54:45.350 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-13T23:54:45.367 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-13T23:54:45.408 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-13T23:54:45.670 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-routes. 2023-10-13T23:54:45.696 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-13T23:54:45.736 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-13T23:54:46.050 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-13T23:54:46.077 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:46.116 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:46.571 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-13T23:54:46.597 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-13T23:54:46.637 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-13T23:54:47.167 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-joblib. 2023-10-13T23:54:47.194 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-13T23:54:47.241 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-13T23:54:47.588 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-13T23:54:47.614 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-13T23:54:47.654 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-13T23:54:47.966 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-sklearn. 2023-10-13T23:54:47.993 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-13T23:54:48.032 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-13T23:54:48.768 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-13T23:54:48.795 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:48.834 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:49.821 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-cachetools. 2023-10-13T23:54:49.848 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-13T23:54:49.888 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-13T23:54:50.191 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-13T23:54:50.218 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-13T23:54:50.266 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-13T23:54:50.854 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-13T23:54:50.880 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-13T23:54:50.920 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-13T23:54:51.240 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-rsa. 2023-10-13T23:54:51.267 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-13T23:54:51.307 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-13T23:54:51.628 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-google-auth. 2023-10-13T23:54:51.654 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-13T23:54:51.694 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-13T23:54:52.014 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-13T23:54:52.041 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-13T23:54:52.081 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-13T23:54:52.553 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-websocket. 2023-10-13T23:54:52.581 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-13T23:54:52.628 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-13T23:54:53.007 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-13T23:54:53.034 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-13T23:54:53.082 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-13T23:54:53.764 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-13T23:54:53.791 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:53.830 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:54.100 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-13T23:54:54.116 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-13T23:54:54.159 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-13T23:54:54.504 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-13T23:54:54.531 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-13T23:54:54.570 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-13T23:54:54.840 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package jq. 2023-10-13T23:54:54.856 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-13T23:54:54.898 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-13T23:54:55.110 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package socat. 2023-10-13T23:54:55.125 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-13T23:54:55.160 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-13T23:54:55.480 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package xmlstarlet. 2023-10-13T23:54:55.496 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-13T23:54:55.538 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-13T23:54:55.919 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-test. 2023-10-13T23:54:55.946 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:56.019 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-test (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:58.307 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package ceph-volume. 2023-10-13T23:54:58.334 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-13T23:54:58.373 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking ceph-volume (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:58.745 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-13T23:54:58.761 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:54:58.803 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libcephfs-dev (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:54:59.123 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package nvme-cli. 2023-10-13T23:54:59.139 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-13T23:54:59.181 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-13T23:54:59.530 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-13T23:54:59.546 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-13T23:54:59.596 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-13T23:55:00.017 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-13T23:55:00.044 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-13T23:55:00.083 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-13T23:55:00.545 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pastescript. 2023-10-13T23:55:00.572 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-13T23:55:00.611 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-13T23:55:00.924 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pluggy. 2023-10-13T23:55:00.951 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-13T23:55:00.990 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-13T23:55:01.311 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-psutil. 2023-10-13T23:55:01.328 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-13T23:55:01.369 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-13T23:55:01.715 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-py. 2023-10-13T23:55:01.742 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-13T23:55:01.781 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-13T23:55:02.085 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pygments. 2023-10-13T23:55:02.112 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-13T23:55:02.151 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-13T23:55:02.615 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-13T23:55:02.642 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-13T23:55:02.681 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-13T23:55:02.993 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-toml. 2023-10-13T23:55:03.020 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-13T23:55:03.068 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-13T23:55:03.338 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-pytest. 2023-10-13T23:55:03.365 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-13T23:55:03.396 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-13T23:55:03.758 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package python3-simplejson. 2023-10-13T23:55:03.786 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-13T23:55:03.825 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-13T23:55:04.112 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-13T23:55:04.140 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-13T23:55:04.178 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-13T23:55:04.858 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package radosgw. 2023-10-13T23:55:04.885 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../107-radosgw_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:55:04.924 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking radosgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:05.728 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package rbd-fuse. 2023-10-13T23:55:05.756 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-13T23:55:05.794 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking rbd-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:06.098 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-13T23:55:06.483 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-13T23:55:06.601 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-13T23:55:06.844 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-13T23:55:07.183 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-13T23:55:07.707 INFO:teuthology.orchestra.run.smithi138.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-13T23:55:07.795 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:08.356 INFO:teuthology.orchestra.run.smithi138.stdout:Adding system user cephadm....done 2023-10-13T23:55:08.503 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-13T23:55:08.763 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-13T23:55:09.014 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-13T23:55:09.140 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-13T23:55:09.400 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-13T23:55:09.652 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-13T23:55:09.753 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-13T23:55:10.046 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-13T23:55:10.407 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-13T23:55:10.666 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-13T23:55:10.926 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-13T23:55:11.178 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-13T23:55:11.412 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-13T23:55:11.664 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-ceph-argparse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:11.923 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-13T23:55:12.049 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-13T23:55:12.317 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-13T23:55:12.427 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-13T23:55:12.778 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-13T23:55:12.905 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-13T23:55:13.441 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-13T23:55:13.750 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-13T23:55:14.428 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-13T23:55:14.679 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-13T23:55:14.805 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-13T23:55:14.932 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-13T23:55:15.233 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-13T23:55:15.635 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-13T23:55:15.861 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-13T23:55:16.230 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-13T23:55:16.866 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-13T23:55:17.126 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-13T23:55:17.227 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:17.503 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-idna (3.3-1) ... 2023-10-13T23:55:17.822 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-13T23:55:18.978 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-13T23:55:19.138 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-13T23:55:19.406 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-13T23:55:19.532 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-13T23:55:19.658 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-13T23:55:19.952 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-13T23:55:20.438 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-13T23:55:20.564 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-13T23:55:20.690 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-13T23:55:21.017 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-13T23:55:21.260 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-13T23:55:21.503 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-13T23:55:21.938 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-13T23:55:22.189 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-13T23:55:22.448 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-13T23:55:22.575 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-13T23:55:22.885 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-13T23:55:22.986 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-13T23:55:23.262 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-13T23:55:23.815 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-13T23:55:24.227 INFO:teuthology.orchestra.run.smithi138.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-13T23:55:24.587 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-13T23:55:24.972 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-13T23:55:25.231 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-13T23:55:25.491 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-13T23:55:25.609 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-13T23:55:25.960 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-13T23:55:26.237 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-13T23:55:26.513 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-13T23:55:26.639 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-13T23:55:26.940 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-13T23:55:27.234 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-13T23:55:27.485 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-13T23:55:27.870 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-13T23:55:28.163 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-13T23:55:28.499 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-13T23:55:28.626 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librados2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:28.737 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librgw2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:28.863 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:28.990 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libcephfs2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:29.116 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libradosstriper1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:29.311 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-13T23:55:29.580 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up librbd1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:29.706 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-modules-core (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:29.832 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:30.045 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-13T23:55:30.045 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-13T23:55:30.584 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libcephfs-dev (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:30.685 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rados (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:30.795 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-13T23:55:31.959 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rbd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:32.060 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up rbd-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:32.278 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-rgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:32.413 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up python3-cephfs (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:32.539 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:33.075 INFO:teuthology.orchestra.run.smithi138.stdout:Adding group ceph....done 2023-10-13T23:55:33.436 INFO:teuthology.orchestra.run.smithi138.stdout:Adding system user ceph....done 2023-10-13T23:55:33.595 INFO:teuthology.orchestra.run.smithi138.stdout:Setting system user ceph properties....done 2023-10-13T23:55:33.607 INFO:teuthology.orchestra.run.smithi138.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-13T23:55:33.752 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-13T23:55:34.123 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-13T23:55:34.634 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-test (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:34.735 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up radosgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:35.211 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-13T23:55:35.211 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-13T23:55:35.713 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-base (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:36.064 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-13T23:55:36.573 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mds (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:36.793 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-13T23:55:36.793 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-13T23:55:37.484 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:37.686 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-13T23:55:37.686 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-13T23:55:38.169 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-osd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:38.513 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-13T23:55:38.513 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-13T23:55:39.055 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:39.181 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:39.332 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mon (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:39.544 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-13T23:55:39.544 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-13T23:55:40.058 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:40.185 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:40.295 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-mgr-dashboard (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:40.454 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up ceph-volume (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-13T23:55:40.782 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-13T23:55:41.919 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-13T23:55:42.390 INFO:teuthology.orchestra.run.smithi138.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-13T23:55:42.393 DEBUG:teuthology.parallel:result is None 2023-10-13T23:55:42.394 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-13T23:55:42.538 DEBUG:teuthology.orchestra.run.smithi138:> dpkg-query -W -f '${Version}' ceph 2023-10-13T23:55:42.559 INFO:teuthology.orchestra.run.smithi138.stdout:18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:55:42.560 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:55:42.560 INFO:teuthology.task.install:The correct ceph version 18.0.0-6724-g01fff6a7-1jammy is installed. 2023-10-13T23:55:42.561 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-13T23:55:42.561 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:42.562 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-13T23:55:42.613 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-13T23:55:42.613 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:42.614 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2023-10-13T23:55:42.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-13T23:55:42.719 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-13T23:55:42.719 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:42.720 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-13T23:55:42.771 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-13T23:55:42.823 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-13T23:55:42.823 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:42.823 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2023-10-13T23:55:42.876 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-13T23:55:42.931 INFO:teuthology.run_tasks:Running task ceph... 2023-10-13T23:55:43.001 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-13T23:55:43.002 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 777 /var/log/ceph 2023-10-13T23:55:43.015 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-13T23:55:43.015 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-13T23:55:43.067 INFO:tasks.ceph:Creating extra log directories... 2023-10-13T23:55:43.068 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-13T23:55:43.132 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-13T23:55:43.132 INFO:tasks.ceph:config {'conf': {'client': {'rbd default features': 125}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-13T23:55:43.132 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257', 'branch': 'wip-dis-testing', 'description': 'rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7427257', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'rbd default features': 125}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': '735f68d7bf91d643813a8420fc2b11441f4bd8c4'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2449, 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f', 'sleep_before_teardown': 0, 'subset': '49/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': '735f68d7bf91d643813a8420fc2b11441f4bd8c4', 'targets': {'smithi138.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIbphirixlHwnOwOOVhbCQiR2axA3wmHarKaDteDBvLWHZeDd90LRhhgzWrE6nKf214ifB1/ugwsfdEyiLumX0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-13_23:35:38', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-13T23:55:43.133 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-13T23:55:43.173 DEBUG:teuthology.orchestra.run.smithi138:> sudo install -d -m0777 -- /var/run/ceph 2023-10-13T23:55:43.229 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:43.229 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/scratch_devs of=/dev/stdout 2023-10-13T23:55:43.277 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-13T23:55:43.277 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_1 2023-10-13T23:55:43.324 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-13T23:55:43.325 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-13T23:55:43.325 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 878 Links: 1 2023-10-13T23:55:43.325 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-13T23:55:43.325 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2023-10-13 23:50:35.121654376 +0000 2023-10-13T23:55:43.326 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2023-10-13 23:50:34.753657662 +0000 2023-10-13T23:55:43.326 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2023-10-13 23:50:34.753657662 +0000 2023-10-13T23:55:43.326 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2023-10-13T23:55:43.326 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-13T23:55:43.379 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2023-10-13T23:55:43.379 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2023-10-13T23:55:43.379 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.00044258 s, 1.2 MB/s 2023-10-13T23:55:43.380 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-13T23:55:43.430 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_2 2023-10-13T23:55:43.477 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-13T23:55:43.477 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-13T23:55:43.477 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 896 Links: 1 2023-10-13T23:55:43.478 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-13T23:55:43.478 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2023-10-13 23:50:35.337652444 +0000 2023-10-13T23:55:43.478 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2023-10-13 23:50:35.333652481 +0000 2023-10-13T23:55:43.478 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2023-10-13 23:50:35.333652481 +0000 2023-10-13T23:55:43.479 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2023-10-13T23:55:43.479 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-13T23:55:43.531 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2023-10-13T23:55:43.531 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2023-10-13T23:55:43.531 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000409827 s, 1.2 MB/s 2023-10-13T23:55:43.532 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-13T23:55:43.582 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_3 2023-10-13T23:55:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-13T23:55:43.629 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-13T23:55:43.629 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 917 Links: 1 2023-10-13T23:55:43.630 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-13T23:55:43.630 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2023-10-13 23:50:36.481642227 +0000 2023-10-13T23:55:43.630 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2023-10-13 23:50:35.949646981 +0000 2023-10-13T23:55:43.631 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2023-10-13 23:50:35.949646981 +0000 2023-10-13T23:55:43.631 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2023-10-13T23:55:43.631 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-13T23:55:43.682 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2023-10-13T23:55:43.683 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2023-10-13T23:55:43.683 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000336913 s, 1.5 MB/s 2023-10-13T23:55:43.684 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-13T23:55:43.734 DEBUG:teuthology.orchestra.run.smithi138:> stat /dev/vg_nvme/lv_4 2023-10-13T23:55:43.782 INFO:teuthology.orchestra.run.smithi138.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-13T23:55:43.783 INFO:teuthology.orchestra.run.smithi138.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-13T23:55:43.783 INFO:teuthology.orchestra.run.smithi138.stdout:Device: 5h/5d Inode: 936 Links: 1 2023-10-13T23:55:43.783 INFO:teuthology.orchestra.run.smithi138.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-13T23:55:43.783 INFO:teuthology.orchestra.run.smithi138.stdout:Access: 2023-10-13 23:50:37.053637121 +0000 2023-10-13T23:55:43.783 INFO:teuthology.orchestra.run.smithi138.stdout:Modify: 2023-10-13 23:50:36.725640048 +0000 2023-10-13T23:55:43.784 INFO:teuthology.orchestra.run.smithi138.stdout:Change: 2023-10-13 23:50:36.725640048 +0000 2023-10-13T23:55:43.784 INFO:teuthology.orchestra.run.smithi138.stdout: Birth: - 2023-10-13T23:55:43.784 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-13T23:55:43.834 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records in 2023-10-13T23:55:43.835 INFO:teuthology.orchestra.run.smithi138.stderr:1+0 records out 2023-10-13T23:55:43.835 INFO:teuthology.orchestra.run.smithi138.stderr:512 bytes copied, 0.000332613 s, 1.5 MB/s 2023-10-13T23:55:43.836 DEBUG:teuthology.orchestra.run.smithi138:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-13T23:55:43.887 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-13T23:55:43.887 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-13T23:55:43.887 INFO:tasks.ceph:Generating config... 2023-10-13T23:55:43.889 INFO:tasks.ceph:[client] rbd default features = 125 2023-10-13T23:55:43.889 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-13T23:55:43.889 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-13T23:55:43.889 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-13T23:55:43.890 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-13T23:55:43.890 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-13T23:55:43.890 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-13T23:55:43.890 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-13T23:55:43.891 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-13T23:55:43.891 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-13T23:55:43.891 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-13T23:55:43.891 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2023-10-13T23:55:43.892 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-13T23:55:43.892 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-13T23:55:43.892 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-13T23:55:43.892 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-13T23:55:43.893 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-13T23:55:43.893 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-13T23:55:43.893 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-13T23:55:43.893 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-13T23:55:43.894 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-13T23:55:43.894 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-13T23:55:43.894 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-13T23:55:43.894 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-13T23:55:43.895 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-13T23:55:43.895 INFO:tasks.ceph:Setting up mon.a... 2023-10-13T23:55:43.895 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-13T23:55:43.959 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.keyring 2023-10-13T23:55:43.969 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-13T23:55:44.046 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-13T23:55:44.100 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.138')] 2023-10-13T23:55:44.101 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.138', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default features': 125}, 'mon.a': {}} 2023-10-13T23:55:44.102 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:44.102 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-13T23:55:44.148 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.138 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-13T23:55:44.226 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-13T23:55:44.227 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: generated fsid be73fd16-7c0f-4418-ab81-68a6185f41a8 2023-10-13T23:55:44.227 INFO:teuthology.orchestra.run.smithi138.stdout:setting min_mon_release = pacific 2023-10-13T23:55:44.227 INFO:teuthology.orchestra.run.smithi138.stdout:epoch 0 2023-10-13T23:55:44.228 INFO:teuthology.orchestra.run.smithi138.stdout:fsid be73fd16-7c0f-4418-ab81-68a6185f41a8 2023-10-13T23:55:44.228 INFO:teuthology.orchestra.run.smithi138.stdout:last_changed 2023-10-13T23:55:44.228032+0000 2023-10-13T23:55:44.228 INFO:teuthology.orchestra.run.smithi138.stdout:created 2023-10-13T23:55:44.228032+0000 2023-10-13T23:55:44.228 INFO:teuthology.orchestra.run.smithi138.stdout:min_mon_release 16 (pacific) 2023-10-13T23:55:44.228 INFO:teuthology.orchestra.run.smithi138.stdout:election_strategy: 1 2023-10-13T23:55:44.228 INFO:teuthology.orchestra.run.smithi138.stdout:0: [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] mon.a 2023-10-13T23:55:44.229 INFO:teuthology.orchestra.run.smithi138.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-13T23:55:44.230 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-13T23:55:44.277 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID be73fd16-7c0f-4418-ab81-68a6185f41a8... 2023-10-13T23:55:44.278 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-13T23:55:44.351 INFO:teuthology.orchestra.run.smithi138.stdout:[global] 2023-10-13T23:55:44.351 INFO:teuthology.orchestra.run.smithi138.stdout: chdir = "" 2023-10-13T23:55:44.351 INFO:teuthology.orchestra.run.smithi138.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-13T23:55:44.351 INFO:teuthology.orchestra.run.smithi138.stdout: auth supported = cephx 2023-10-13T23:55:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: filestore xattr use omap = true 2023-10-13T23:55:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: mon clock drift allowed = 1.000 2023-10-13T23:55:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.352 INFO:teuthology.orchestra.run.smithi138.stdout: osd crush chooseleaf type = 0 2023-10-13T23:55:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: auth debug = true 2023-10-13T23:55:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on old message = true 2023-10-13T23:55:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: ms die on bug = true 2023-10-13T23:55:44.353 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-13T23:55:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: mon pg warn max object skew = 0 2023-10-13T23:55:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: # disable pg_autoscaler by default for new pools 2023-10-13T23:55:44.354 INFO:teuthology.orchestra.run.smithi138.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-13T23:55:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default size = 2 2023-10-13T23:55:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow primary affinity = true 2023-10-13T23:55:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd allow pg remap = true 2023-10-13T23:55:44.355 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on legacy crush tunables = false 2023-10-13T23:55:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on crush straw calc version zero = false 2023-10-13T23:55:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on no sortbitwise = false 2023-10-13T23:55:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on osd down out interval zero = false 2023-10-13T23:55:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on too few osds = false 2023-10-13T23:55:44.356 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-13T23:55:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-13T23:55:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: mon_allow_pool_size_one = true 2023-10-13T23:55:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-13T23:55:44.357 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: osd default data pool replay window = 5 2023-10-13T23:55:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: mon allow pool delete = true 2023-10-13T23:55:44.358 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: mon cluster log file level = debug 2023-10-13T23:55:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: debug asserts on shutdown = true 2023-10-13T23:55:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: mon health detail to clog = false 2023-10-13T23:55:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: mon host = 172.21.15.138 2023-10-13T23:55:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: mon client directed command retry = 5 2023-10-13T23:55:44.359 INFO:teuthology.orchestra.run.smithi138.stdout: mon warn on pool no app = False 2023-10-13T23:55:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: ms inject socket failures = 5000 2023-10-13T23:55:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: fsid = be73fd16-7c0f-4418-ab81-68a6185f41a8 2023-10-13T23:55:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.360 INFO:teuthology.orchestra.run.smithi138.stdout:[osd] 2023-10-13T23:55:44.360 INFO:teuthology.orchestra.run.smithi138.stdout: osd journal size = 100 2023-10-13T23:55:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub load threshold = 5.0 2023-10-13T23:55:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: osd scrub max interval = 600 2023-10-13T23:55:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: osd mclock profile = high_recovery_ops 2023-10-13T23:55:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.361 INFO:teuthology.orchestra.run.smithi138.stdout: osd recover clone overlap = true 2023-10-13T23:55:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: osd recovery max chunk = 1048576 2023-10-13T23:55:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug shutdown = true 2023-10-13T23:55:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug op order = true 2023-10-13T23:55:44.362 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug verify stray on activate = true 2023-10-13T23:55:44.363 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.363 INFO:teuthology.orchestra.run.smithi138.stdout: osd open classes on start = true 2023-10-13T23:55:44.363 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug pg log writeout = true 2023-10-13T23:55:44.363 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.363 INFO:teuthology.orchestra.run.smithi138.stdout: osd deep scrub update digest min age = 30 2023-10-13T23:55:44.363 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.364 INFO:teuthology.orchestra.run.smithi138.stdout: osd map max advance = 10 2023-10-13T23:55:44.364 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.364 INFO:teuthology.orchestra.run.smithi138.stdout: journal zero on create = true 2023-10-13T23:55:44.364 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.364 INFO:teuthology.orchestra.run.smithi138.stdout: filestore ondisk finisher threads = 3 2023-10-13T23:55:44.364 INFO:teuthology.orchestra.run.smithi138.stdout: filestore apply finisher threads = 3 2023-10-13T23:55:44.365 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.365 INFO:teuthology.orchestra.run.smithi138.stdout: bdev debug aio = true 2023-10-13T23:55:44.365 INFO:teuthology.orchestra.run.smithi138.stdout: osd debug misdirected ops = true 2023-10-13T23:55:44.365 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore block size = 96636764160 2023-10-13T23:55:44.365 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression algorithm = lz4 2023-10-13T23:55:44.366 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore compression mode = aggressive 2023-10-13T23:55:44.366 INFO:teuthology.orchestra.run.smithi138.stdout: bluestore fsck on mount = True 2023-10-13T23:55:44.366 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluefs = 1/20 2023-10-13T23:55:44.366 INFO:teuthology.orchestra.run.smithi138.stdout: debug bluestore = 1/20 2023-10-13T23:55:44.366 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2023-10-13T23:55:44.367 INFO:teuthology.orchestra.run.smithi138.stdout: debug osd = 20 2023-10-13T23:55:44.367 INFO:teuthology.orchestra.run.smithi138.stdout: debug rocksdb = 4/10 2023-10-13T23:55:44.367 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd backfillfull_ratio = 0.85 2023-10-13T23:55:44.367 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd full ratio = 0.9 2023-10-13T23:55:44.367 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd nearfull ratio = 0.8 2023-10-13T23:55:44.367 INFO:teuthology.orchestra.run.smithi138.stdout: osd failsafe full ratio = 0.95 2023-10-13T23:55:44.368 INFO:teuthology.orchestra.run.smithi138.stdout: osd objectstore = bluestore 2023-10-13T23:55:44.368 INFO:teuthology.orchestra.run.smithi138.stdout: osd shutdown pgref assert = True 2023-10-13T23:55:44.368 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.368 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr] 2023-10-13T23:55:44.368 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2023-10-13T23:55:44.369 INFO:teuthology.orchestra.run.smithi138.stdout: debug mgr = 20 2023-10-13T23:55:44.369 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2023-10-13T23:55:44.369 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2023-10-13T23:55:44.369 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2023-10-13T23:55:44.369 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2023-10-13T23:55:44.369 INFO:teuthology.orchestra.run.smithi138.stdout: mgr/telemetry/nag = false 2023-10-13T23:55:44.370 INFO:teuthology.orchestra.run.smithi138.stdout: debug rbd = 20 2023-10-13T23:55:44.370 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.370 INFO:teuthology.orchestra.run.smithi138.stdout:[mon] 2023-10-13T23:55:44.370 INFO:teuthology.orchestra.run.smithi138.stdout: debug ms = 1 2023-10-13T23:55:44.370 INFO:teuthology.orchestra.run.smithi138.stdout: debug mon = 20 2023-10-13T23:55:44.371 INFO:teuthology.orchestra.run.smithi138.stdout: debug paxos = 20 2023-10-13T23:55:44.371 INFO:teuthology.orchestra.run.smithi138.stdout: debug auth = 20 2023-10-13T23:55:44.371 INFO:teuthology.orchestra.run.smithi138.stdout: mon data avail warn = 5 2023-10-13T23:55:44.371 INFO:teuthology.orchestra.run.smithi138.stdout: mon mgr mkfs grace = 240 2023-10-13T23:55:44.371 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min pgs per osd = 4 2023-10-13T23:55:44.371 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd reporter subtree level = osd 2023-10-13T23:55:44.372 INFO:teuthology.orchestra.run.smithi138.stdout: mon osd prime pg temp = true 2023-10-13T23:55:44.372 INFO:teuthology.orchestra.run.smithi138.stdout: mon reweight min bytes per osd = 10 2023-10-13T23:55:44.372 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.372 INFO:teuthology.orchestra.run.smithi138.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-13T23:55:44.372 INFO:teuthology.orchestra.run.smithi138.stdout: auth mon ticket ttl = 660 # 11m 2023-10-13T23:55:44.372 INFO:teuthology.orchestra.run.smithi138.stdout: auth service ticket ttl = 240 # 4m 2023-10-13T23:55:44.373 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.373 INFO:teuthology.orchestra.run.smithi138.stdout: # don't complain about insecure global_id in the test suite 2023-10-13T23:55:44.373 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-13T23:55:44.373 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-13T23:55:44.373 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.374 INFO:teuthology.orchestra.run.smithi138.stdout: # 1m isn't quite enough 2023-10-13T23:55:44.374 INFO:teuthology.orchestra.run.smithi138.stdout: mon_down_mkfs_grace = 2m 2023-10-13T23:55:44.374 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.374 INFO:teuthology.orchestra.run.smithi138.stdout: mon_warn_on_filestore_osds = false 2023-10-13T23:55:44.374 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:44.374 INFO:teuthology.orchestra.run.smithi138.stdout:[client] 2023-10-13T23:55:44.375 INFO:teuthology.orchestra.run.smithi138.stdout: rgw cache enabled = true 2023-10-13T23:55:44.375 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable ops log = true 2023-10-13T23:55:44.375 INFO:teuthology.orchestra.run.smithi138.stdout: rgw enable usage log = true 2023-10-13T23:55:44.375 INFO:teuthology.orchestra.run.smithi138.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-13T23:55:44.375 INFO:teuthology.orchestra.run.smithi138.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-13T23:55:44.376 INFO:teuthology.orchestra.run.smithi138.stdout: rbd default features = 125 2023-10-13T23:55:44.376 INFO:teuthology.orchestra.run.smithi138.stdout:[mon.a] 2023-10-13T23:55:44.377 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-13T23:55:44.377 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-13T23:55:44.467 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-13T23:55:44.468 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:44.468 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-13T23:55:44.473 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:44.473 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-13T23:55:44.520 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi138.front.sepia.ceph.com 2023-10-13T23:55:44.520 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:44.521 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-13T23:55:44.521 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-13T23:55:44.584 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:44.584 DEBUG:teuthology.orchestra.run.smithi138:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-13T23:55:44.632 INFO:tasks.ceph:Setting up mon nodes... 2023-10-13T23:55:44.633 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-13T23:55:44.633 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-13T23:55:44.718 INFO:teuthology.orchestra.run.smithi138.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-13T23:55:44.721 INFO:tasks.ceph:Setting up mds nodes... 2023-10-13T23:55:44.721 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-13T23:55:44.722 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-13T23:55:44.795 INFO:teuthology.orchestra.run.smithi138.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-13T23:55:44.806 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-13T23:55:44.806 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi138.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-13T23:55:44.807 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-13T23:55:44.863 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-13T23:55:44.863 INFO:tasks.ceph:role: osd.0 2023-10-13T23:55:44.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com 2023-10-13T23:55:44.864 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-13T23:55:44.923 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-13T23:55:44.923 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-13T23:55:44.924 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-13T23:55:44.924 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-13T23:55:44.924 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-13T23:55:44.924 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2023-10-13T23:55:44.924 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-13T23:55:44.925 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-13T23:55:44.925 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-13T23:55:44.925 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-13T23:55:45.377 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2023-10-13T23:55:45.379 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2023-10-13T23:55:45.379 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-13T23:55:45.464 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-13T23:55:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-13T23:55:45.519 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: /sbin/restorecon: command not found 2023-10-13T23:55:45.519 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-13T23:55:45.576 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-13T23:55:45.576 INFO:tasks.ceph:role: osd.1 2023-10-13T23:55:45.576 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com 2023-10-13T23:55:45.577 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-13T23:55:45.641 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-13T23:55:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-13T23:55:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-13T23:55:45.642 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-13T23:55:45.642 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-13T23:55:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2023-10-13T23:55:45.643 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-13T23:55:45.643 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-13T23:55:45.643 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-13T23:55:45.643 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-13T23:55:46.146 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2023-10-13T23:55:46.149 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2023-10-13T23:55:46.149 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-13T23:55:46.282 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-13T23:55:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-13T23:55:46.294 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: /sbin/restorecon: command not found 2023-10-13T23:55:46.295 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-13T23:55:46.347 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-13T23:55:46.348 INFO:tasks.ceph:role: osd.2 2023-10-13T23:55:46.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com 2023-10-13T23:55:46.349 DEBUG:teuthology.orchestra.run.smithi138:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-13T23:55:46.412 INFO:teuthology.orchestra.run.smithi138.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-13T23:55:46.412 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-13T23:55:46.412 INFO:teuthology.orchestra.run.smithi138.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-13T23:55:46.412 INFO:teuthology.orchestra.run.smithi138.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-13T23:55:46.413 INFO:teuthology.orchestra.run.smithi138.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-13T23:55:46.413 INFO:teuthology.orchestra.run.smithi138.stdout: = sunit=0 swidth=0 blks 2023-10-13T23:55:46.413 INFO:teuthology.orchestra.run.smithi138.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-13T23:55:46.413 INFO:teuthology.orchestra.run.smithi138.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-13T23:55:46.413 INFO:teuthology.orchestra.run.smithi138.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-13T23:55:46.414 INFO:teuthology.orchestra.run.smithi138.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-13T23:55:46.885 INFO:teuthology.orchestra.run.smithi138.stdout:Discarding blocks...Done. 2023-10-13T23:55:46.887 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi138.front.sepia.ceph.com -o noatime 2023-10-13T23:55:46.887 DEBUG:teuthology.orchestra.run.smithi138:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-13T23:55:47.023 DEBUG:teuthology.orchestra.run.smithi138:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-13T23:55:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-13T23:55:47.035 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: /sbin/restorecon: command not found 2023-10-13T23:55:47.036 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-13T23:55:47.153 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:47.151+0000 7f6bb532d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-13T23:55:47.154 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:47.151+0000 7f6bb532d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-13T23:55:47.154 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:47.151+0000 7f6bb532d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-13T23:55:47.155 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:47.151+0000 7f6bb532d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-13T23:55:47.155 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:47.151+0000 7f6bb532d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-13T23:55:47.156 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:47.155+0000 7f6bb532d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-13T23:55:50.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-13T23:55:50.191 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-13T23:55:50.270 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:50.267+0000 7f1705455640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-13T23:55:50.271 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:50.267+0000 7f1705455640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-13T23:55:50.271 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:50.267+0000 7f1705455640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-13T23:55:50.272 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:50.267+0000 7f1705455640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-13T23:55:50.272 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:50.267+0000 7f1705455640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-13T23:55:50.272 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:50.271+0000 7f1705455640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-13T23:55:53.295 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-13T23:55:53.350 DEBUG:teuthology.orchestra.run.smithi138:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-13T23:55:53.432 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:53.431+0000 7f0312036640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-13T23:55:53.433 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:53.431+0000 7f0312036640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-13T23:55:53.433 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:53.431+0000 7f0312036640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-13T23:55:53.433 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:53.431+0000 7f0312036640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-13T23:55:53.434 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:53.431+0000 7f0312036640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-13T23:55:53.434 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-13T23:55:53.431+0000 7f0312036640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-13T23:55:56.439 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-13T23:55:56.492 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-13T23:55:56.492 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:56.492 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-13T23:55:56.546 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:56.547 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-13T23:55:56.599 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:56.599 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-13T23:55:56.650 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:56.650 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-13T23:55:56.702 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:56.702 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-13T23:55:56.748 INFO:tasks.ceph:Adding keys to all mons... 2023-10-13T23:55:56.748 DEBUG:teuthology.orchestra.run.smithi138:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-13T23:55:56.801 INFO:teuthology.orchestra.run.smithi138.stdout:[mgr.x] 2023-10-13T23:55:56.802 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQAA2Sll///uKhAAplMQw09os6aSXBpuG7xDxg== 2023-10-13T23:55:56.802 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.0] 2023-10-13T23:55:56.802 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQAD2SllEdU6CRAAFHBRYHOxjfGdxdF5z3Vozg== 2023-10-13T23:55:56.802 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.1] 2023-10-13T23:55:56.803 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQAG2SllPek5EBAAtFdysI89roU+ho6W5iiefg== 2023-10-13T23:55:56.803 INFO:teuthology.orchestra.run.smithi138.stdout:[osd.2] 2023-10-13T23:55:56.803 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQAJ2SlldrLaGRAAmUuYW64tOK60KVSZ6C8Xnw== 2023-10-13T23:55:56.803 INFO:teuthology.orchestra.run.smithi138.stdout:[client.0] 2023-10-13T23:55:56.803 INFO:teuthology.orchestra.run.smithi138.stdout: key = AQAA2SllVQh5LxAAUpCEox2uoU1dTd1rKKZ/oQ== 2023-10-13T23:55:56.804 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-13T23:55:56.894 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-13T23:55:56.976 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-13T23:55:57.063 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-13T23:55:57.152 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-13T23:55:57.239 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-13T23:55:57.240 DEBUG:teuthology.orchestra.run.smithi138:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-13T23:55:57.295 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-13T23:55:57.394 DEBUG:teuthology.orchestra.run.smithi138:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-13T23:55:57.405 DEBUG:teuthology.orchestra.run.smithi138:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-13T23:55:57.451 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-13T23:55:57.452 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-13T23:55:57.453 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-13T23:55:57.501 INFO:tasks.ceph.mon.a:Started 2023-10-13T23:55:57.501 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-13T23:55:57.502 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-13T23:55:57.505 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-13T23:55:57.507 INFO:tasks.ceph.mgr.x:Started 2023-10-13T23:55:57.508 INFO:tasks.ceph:Setting crush tunables to default 2023-10-13T23:55:57.508 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd crush tunables default 2023-10-13T23:55:57.759 INFO:teuthology.orchestra.run.smithi138.stderr:adjusted tunables profile to default 2023-10-13T23:55:57.777 INFO:tasks.ceph:check_enable_crimson: False 2023-10-13T23:55:57.777 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-13T23:55:57.778 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:57.778 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-13T23:55:57.789 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:57.789 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-13T23:55:57.841 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:55:57.842 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-13T23:55:57.847 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:57.843+0000 7f088cbb2180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-13T23:55:57.857 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new a5f50877-3e63-4f7c-a58b-da90780b0e61 0 2023-10-13T23:55:58.094 INFO:teuthology.orchestra.run.smithi138.stdout:0 2023-10-13T23:55:58.112 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new 6978755c-6c5b-4d64-a2de-bac99a43601c 1 2023-10-13T23:55:58.305 INFO:teuthology.orchestra.run.smithi138.stdout:1 2023-10-13T23:55:58.321 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd new c105e221-238a-4079-8e50-7ac981476525 2 2023-10-13T23:55:58.375 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:58.371+0000 7f088cbb2180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-13T23:55:58.541 INFO:teuthology.orchestra.run.smithi138.stdout:2 2023-10-13T23:55:58.564 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-13T23:55:58.564 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-13T23:55:58.566 INFO:tasks.ceph.osd.0:Started 2023-10-13T23:55:58.567 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-13T23:55:58.567 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-13T23:55:58.569 INFO:tasks.ceph.osd.1:Started 2023-10-13T23:55:58.570 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-13T23:55:58.570 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-13T23:55:58.572 INFO:tasks.ceph.osd.2:Started 2023-10-13T23:55:58.573 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-13T23:55:58.597 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:58.595+0000 7f088cbb2180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-13T23:55:58.750 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:55:58.750 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":5,"fsid":"be73fd16-7c0f-4418-ab81-68a6185f41a8","created":"2023-10-13T23:55:57.637104+0000","modified":"2023-10-13T23:55:58.541401+0000","last_up_change":"0.000000","last_in_change":"2023-10-13T23:55:58.541401+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5f50877-3e63-4f7c-a58b-da90780b0e61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6978755c-6c5b-4d64-a2de-bac99a43601c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c105e221-238a-4079-8e50-7ac981476525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-13T23:55:58.766 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-13T23:55:58.766 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-13T23:55:59.068 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-13T23:55:59.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:55:59.167+0000 7feba2426640 -1 Falling back to public interface 2023-10-13T23:55:59.169 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-13T23:55:59.167+0000 7f524719c640 -1 Falling back to public interface 2023-10-13T23:55:59.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:55:59.199+0000 7f23e480c640 -1 Falling back to public interface 2023-10-13T23:55:59.218 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.215+0000 7f088cbb2180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-13T23:55:59.245 INFO:teuthology.misc.health.smithi138.stdout: 2023-10-13T23:55:59.246 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":5,"fsid":"be73fd16-7c0f-4418-ab81-68a6185f41a8","created":"2023-10-13T23:55:57.637104+0000","modified":"2023-10-13T23:55:58.541401+0000","last_up_change":"0.000000","last_in_change":"2023-10-13T23:55:58.541401+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5f50877-3e63-4f7c-a58b-da90780b0e61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6978755c-6c5b-4d64-a2de-bac99a43601c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c105e221-238a-4079-8e50-7ac981476525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-13T23:55:59.260 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-13T23:55:59.273 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.271+0000 7f088cbb2180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-13T23:55:59.325 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.323+0000 7f088cbb2180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-13T23:55:59.401 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.399+0000 7f088cbb2180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-13T23:55:59.453 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.451+0000 7f088cbb2180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-13T23:55:59.529 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.527+0000 7f088cbb2180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-13T23:55:59.580 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.575+0000 7f088cbb2180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-13T23:55:59.718 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.715+0000 7f088cbb2180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-13T23:55:59.777 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:55:59.775+0000 7f088cbb2180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-13T23:56:00.438 INFO:tasks.ceph.mgr.x.smithi138.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-13T23:56:00.438 INFO:tasks.ceph.mgr.x.smithi138.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-13T23:56:00.439 INFO:tasks.ceph.mgr.x.smithi138.stderr: from numpy import show_config as show_numpy_config 2023-10-13T23:56:00.490 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:00.487+0000 7f088cbb2180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-13T23:56:00.629 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:00.627+0000 7f088cbb2180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-13T23:56:00.798 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:00.795+0000 7f088cbb2180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-13T23:56:00.867 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:00.863+0000 7f088cbb2180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-13T23:56:00.995 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:00.991+0000 7f088cbb2180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-13T23:56:01.587 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:01.583+0000 7f088cbb2180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-13T23:56:01.662 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:01.659+0000 7f088cbb2180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-13T23:56:01.837 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:01.835+0000 7f088cbb2180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-13T23:56:01.909 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:01.907+0000 7f088cbb2180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-13T23:56:02.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:02.219+0000 7f088cbb2180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-13T23:56:02.283 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:02.279+0000 7f088cbb2180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-13T23:56:02.818 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-13T23:56:02.815+0000 7f524719c640 -1 osd.0 0 log_to_monitors true 2023-10-13T23:56:02.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:56:02.819+0000 7f23e480c640 -1 osd.1 0 log_to_monitors true 2023-10-13T23:56:03.109 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:56:03.107+0000 7feba2426640 -1 osd.2 0 log_to_monitors true 2023-10-13T23:56:05.341 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:56:05.339+0000 7f0834329640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-13T23:56:05.563 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-13T23:56:05.675 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-13T23:56:05.671+0000 7f5243958640 -1 osd.0 0 waiting for initial osdmap 2023-10-13T23:56:05.712 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-13T23:56:05.711+0000 7f523af79640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-13T23:56:05.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:56:05.727+0000 7feb9e3ce640 -1 osd.2 0 waiting for initial osdmap 2023-10-13T23:56:05.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:56:05.763+0000 7f23e0fc8640 -1 osd.1 0 waiting for initial osdmap 2023-10-13T23:56:05.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:56:05.771+0000 7feb96203640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-13T23:56:05.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:56:05.787+0000 7f23d85e9640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-13T23:56:05.881 INFO:teuthology.misc.health.smithi138.stdout: 2023-10-13T23:56:05.881 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":7,"fsid":"be73fd16-7c0f-4418-ab81-68a6185f41a8","created":"2023-10-13T23:55:57.637104+0000","modified":"2023-10-13T23:56:04.589871+0000","last_up_change":"0.000000","last_in_change":"2023-10-13T23:55:58.541401+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5f50877-3e63-4f7c-a58b-da90780b0e61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6978755c-6c5b-4d64-a2de-bac99a43601c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c105e221-238a-4079-8e50-7ac981476525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-13T23:56:05.896 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-13T23:56:12.199 DEBUG:teuthology.orchestra.run.smithi138:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-13T23:56:12.492 INFO:teuthology.misc.health.smithi138.stdout: 2023-10-13T23:56:12.492 INFO:teuthology.misc.health.smithi138.stdout:{"epoch":12,"fsid":"be73fd16-7c0f-4418-ab81-68a6185f41a8","created":"2023-10-13T23:55:57.637104+0000","modified":"2023-10-13T23:56:10.710811+0000","last_up_change":"2023-10-13T23:56:06.677931+0000","last_in_change":"2023-10-13T23:55:58.541401+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-13T23:56:08.413739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5f50877-3e63-4f7c-a58b-da90780b0e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6807","nonce":4261051765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6811","nonce":4261051765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6815","nonce":4261051765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6813","nonce":4261051765}]},"public_addr":"172.21.15.138:6807/4261051765","cluster_addr":"172.21.15.138:6811/4261051765","heartbeat_back_addr":"172.21.15.138:6815/4261051765","heartbeat_front_addr":"172.21.15.138:6813/4261051765","state":["exists","up"]},{"osd":1,"uuid":"6978755c-6c5b-4d64-a2de-bac99a43601c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6817","nonce":4204819583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6819","nonce":4204819583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6823","nonce":4204819583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6821","nonce":4204819583}]},"public_addr":"172.21.15.138:6817/4204819583","cluster_addr":"172.21.15.138:6819/4204819583","heartbeat_back_addr":"172.21.15.138:6823/4204819583","heartbeat_front_addr":"172.21.15.138:6821/4204819583","state":["exists","up"]},{"osd":2,"uuid":"c105e221-238a-4079-8e50-7ac981476525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6801","nonce":2835165467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6803","nonce":2835165467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6810","nonce":2835165467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6806","nonce":2835165467}]},"public_addr":"172.21.15.138:6801/2835165467","cluster_addr":"172.21.15.138:6803/2835165467","heartbeat_back_addr":"172.21.15.138:6810/2835165467","heartbeat_front_addr":"172.21.15.138:6806/2835165467","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.800709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.799864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:04.109910+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-13T23:56:12.509 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-13T23:56:12.510 INFO:tasks.ceph:Creating RBD pool 2023-10-13T23:56:12.510 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-13T23:56:13.735 INFO:teuthology.orchestra.run.smithi138.stderr:pool 'rbd' created 2023-10-13T23:56:13.764 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-13T23:56:14.742 INFO:teuthology.orchestra.run.smithi138.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-13T23:56:14.769 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-13T23:56:14.770 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-13T23:56:14.770 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-13T23:56:14.770 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-13T23:56:14.771 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-13T23:56:15.083 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:56:15.099 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6824","nonce":2528842459},{"type":"v1","addr":"172.21.15.138:6825","nonce":2528842459}]},"active_addr":"172.21.15.138:6825/2528842459","active_change":"2023-10-13T23:56:04.328057+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3879549450}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":3069854595}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":2213336685}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.138:0","nonce":525565770}]}]} 2023-10-13T23:56:15.102 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-13T23:56:15.102 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-13T23:56:15.102 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-13T23:56:15.394 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:56:15.394 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":14,"fsid":"be73fd16-7c0f-4418-ab81-68a6185f41a8","created":"2023-10-13T23:55:57.637104+0000","modified":"2023-10-13T23:56:14.737101+0000","last_up_change":"2023-10-13T23:56:06.677931+0000","last_in_change":"2023-10-13T23:55:58.541401+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-13T23:56:08.413739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-13T23:56:12.839467+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5f50877-3e63-4f7c-a58b-da90780b0e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6807","nonce":4261051765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6811","nonce":4261051765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6815","nonce":4261051765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6813","nonce":4261051765}]},"public_addr":"172.21.15.138:6807/4261051765","cluster_addr":"172.21.15.138:6811/4261051765","heartbeat_back_addr":"172.21.15.138:6815/4261051765","heartbeat_front_addr":"172.21.15.138:6813/4261051765","state":["exists","up"]},{"osd":1,"uuid":"6978755c-6c5b-4d64-a2de-bac99a43601c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6817","nonce":4204819583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6819","nonce":4204819583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6823","nonce":4204819583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6821","nonce":4204819583}]},"public_addr":"172.21.15.138:6817/4204819583","cluster_addr":"172.21.15.138:6819/4204819583","heartbeat_back_addr":"172.21.15.138:6823/4204819583","heartbeat_front_addr":"172.21.15.138:6821/4204819583","state":["exists","up"]},{"osd":2,"uuid":"c105e221-238a-4079-8e50-7ac981476525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6801","nonce":2835165467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6803","nonce":2835165467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6810","nonce":2835165467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6806","nonce":2835165467}]},"public_addr":"172.21.15.138:6801/2835165467","cluster_addr":"172.21.15.138:6803/2835165467","heartbeat_back_addr":"172.21.15.138:6810/2835165467","heartbeat_front_addr":"172.21.15.138:6806/2835165467","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.800709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.799864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:04.109910+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-13T23:56:15.412 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-13T23:56:15.413 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-13T23:56:15.681 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:56:15.681 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":14,"fsid":"be73fd16-7c0f-4418-ab81-68a6185f41a8","created":"2023-10-13T23:55:57.637104+0000","modified":"2023-10-13T23:56:14.737101+0000","last_up_change":"2023-10-13T23:56:06.677931+0000","last_in_change":"2023-10-13T23:55:58.541401+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-13T23:56:08.413739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-13T23:56:12.839467+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5f50877-3e63-4f7c-a58b-da90780b0e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6807","nonce":4261051765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6811","nonce":4261051765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6815","nonce":4261051765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6813","nonce":4261051765}]},"public_addr":"172.21.15.138:6807/4261051765","cluster_addr":"172.21.15.138:6811/4261051765","heartbeat_back_addr":"172.21.15.138:6815/4261051765","heartbeat_front_addr":"172.21.15.138:6813/4261051765","state":["exists","up"]},{"osd":1,"uuid":"6978755c-6c5b-4d64-a2de-bac99a43601c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6817","nonce":4204819583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6819","nonce":4204819583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6823","nonce":4204819583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6821","nonce":4204819583}]},"public_addr":"172.21.15.138:6817/4204819583","cluster_addr":"172.21.15.138:6819/4204819583","heartbeat_back_addr":"172.21.15.138:6823/4204819583","heartbeat_front_addr":"172.21.15.138:6821/4204819583","state":["exists","up"]},{"osd":2,"uuid":"c105e221-238a-4079-8e50-7ac981476525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6801","nonce":2835165467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6803","nonce":2835165467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6810","nonce":2835165467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6806","nonce":2835165467}]},"public_addr":"172.21.15.138:6801/2835165467","cluster_addr":"172.21.15.138:6803/2835165467","heartbeat_back_addr":"172.21.15.138:6810/2835165467","heartbeat_front_addr":"172.21.15.138:6806/2835165467","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.800709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.799864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:04.109910+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-13T23:56:15.698 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-13T23:56:15.699 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-13T23:56:15.699 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-13T23:56:15.851 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2023-10-13T23:56:15.851 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-13T23:56:15.852 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2023-10-13T23:56:15.853 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-13T23:56:15.854 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2023-10-13T23:56:15.854 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-13T23:56:16.127 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2023-10-13T23:56:16.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.2 2023-10-13T23:56:16.166 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2023-10-13T23:56:16.175 INFO:teuthology.orchestra.run.smithi138.stdout:34359738371 2023-10-13T23:56:16.184 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.0 2023-10-13T23:56:16.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738371 for osd.1 2023-10-13T23:56:17.146 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-13T23:56:17.184 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-13T23:56:17.193 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-13T23:56:17.446 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2023-10-13T23:56:17.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2023-10-13T23:56:17.464 DEBUG:teuthology.parallel:result is None 2023-10-13T23:56:17.489 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2023-10-13T23:56:17.499 INFO:teuthology.orchestra.run.smithi138.stdout:34359738372 2023-10-13T23:56:17.506 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2023-10-13T23:56:17.507 DEBUG:teuthology.parallel:result is None 2023-10-13T23:56:17.517 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2023-10-13T23:56:17.517 DEBUG:teuthology.parallel:result is None 2023-10-13T23:56:17.518 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-13T23:56:17.518 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-13T23:56:17.820 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:56:17.821 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-13T23:56:17.837 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-13T23:56:16.339658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81532,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033988,"statfs":{"total":289910292480,"available":289826803712,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":12496,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.601901"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.761667+0000","last_change":"2023-10-13T23:56:14.757466+0000","last_active":"2023-10-13T23:56:14.761667+0000","last_peered":"2023-10-13T23:56:14.761667+0000","last_clean":"2023-10-13T23:56:14.761667+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-13T23:56:14.761667+0000","last_undegraded":"2023-10-13T23:56:14.761667+0000","last_fullsized":"2023-10-13T23:56:14.761667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:06:12.698773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.759098+0000","last_change":"2023-10-13T23:56:14.755924+0000","last_active":"2023-10-13T23:56:14.759098+0000","last_peered":"2023-10-13T23:56:14.759098+0000","last_clean":"2023-10-13T23:56:14.759098+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-13T23:56:14.759098+0000","last_undegraded":"2023-10-13T23:56:14.759098+0000","last_fullsized":"2023-10-13T23:56:14.759098+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:00:17.635268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.761652+0000","last_change":"2023-10-13T23:56:14.757164+0000","last_active":"2023-10-13T23:56:14.761652+0000","last_peered":"2023-10-13T23:56:14.761652+0000","last_clean":"2023-10-13T23:56:14.761652+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-13T23:56:14.761652+0000","last_undegraded":"2023-10-13T23:56:14.761652+0000","last_fullsized":"2023-10-13T23:56:14.761652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:30:38.094808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.757356+0000","last_change":"2023-10-13T23:56:14.755160+0000","last_active":"2023-10-13T23:56:14.757356+0000","last_peered":"2023-10-13T23:56:14.757356+0000","last_clean":"2023-10-13T23:56:14.757356+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-13T23:56:14.757356+0000","last_undegraded":"2023-10-13T23:56:14.757356+0000","last_fullsized":"2023-10-13T23:56:14.757356+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:50:15.986492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.756230+0000","last_change":"2023-10-13T23:56:14.753469+0000","last_active":"2023-10-13T23:56:14.756230+0000","last_peered":"2023-10-13T23:56:14.756230+0000","last_clean":"2023-10-13T23:56:14.756230+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-13T23:56:14.756230+0000","last_undegraded":"2023-10-13T23:56:14.756230+0000","last_fullsized":"2023-10-13T23:56:14.756230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:16:53.104574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-13T23:56:15.748585+0000","last_change":"2023-10-13T23:56:14.754720+0000","last_active":"2023-10-13T23:56:15.748585+0000","last_peered":"2023-10-13T23:56:15.748585+0000","last_clean":"2023-10-13T23:56:15.748585+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-13T23:56:15.748585+0000","last_undegraded":"2023-10-13T23:56:15.748585+0000","last_fullsized":"2023-10-13T23:56:15.748585+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:39:43.759783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-13T23:56:15.748586+0000","last_change":"2023-10-13T23:56:14.754793+0000","last_active":"2023-10-13T23:56:15.748586+0000","last_peered":"2023-10-13T23:56:15.748586+0000","last_clean":"2023-10-13T23:56:15.748586+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-13T23:56:15.748586+0000","last_undegraded":"2023-10-13T23:56:15.748586+0000","last_fullsized":"2023-10-13T23:56:15.748586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:22:24.150498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.756823+0000","last_change":"2023-10-13T23:56:14.754310+0000","last_active":"2023-10-13T23:56:14.756823+0000","last_peered":"2023-10-13T23:56:14.756823+0000","last_clean":"2023-10-13T23:56:14.756823+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-13T23:56:14.756823+0000","last_undegraded":"2023-10-13T23:56:14.756823+0000","last_fullsized":"2023-10-13T23:56:14.756823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:12:50.830266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-13T23:56:13.747465+0000","last_change":"2023-10-13T23:56:09.871340+0000","last_active":"2023-10-13T23:56:13.747465+0000","last_peered":"2023-10-13T23:56:13.747465+0000","last_clean":"2023-10-13T23:56:13.747465+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-13T23:56:13.747465+0000","last_undegraded":"2023-10-13T23:56:13.747465+0000","last_fullsized":"2023-10-13T23:56:13.747465+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_clean_scrub_stamp":"2023-10-13T23:56:08.695661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:39:25.194520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12496,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-13T23:56:17.838 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-13T23:56:18.103 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:56:18.104 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-13T23:56:18.120 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-13T23:56:16.339658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81532,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033988,"statfs":{"total":289910292480,"available":289826803712,"internally_reserved":0,"allocated":483328,"data_stored":1281896,"data_compressed":12496,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.601901"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.761667+0000","last_change":"2023-10-13T23:56:14.757466+0000","last_active":"2023-10-13T23:56:14.761667+0000","last_peered":"2023-10-13T23:56:14.761667+0000","last_clean":"2023-10-13T23:56:14.761667+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-13T23:56:14.761667+0000","last_undegraded":"2023-10-13T23:56:14.761667+0000","last_fullsized":"2023-10-13T23:56:14.761667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:06:12.698773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.759098+0000","last_change":"2023-10-13T23:56:14.755924+0000","last_active":"2023-10-13T23:56:14.759098+0000","last_peered":"2023-10-13T23:56:14.759098+0000","last_clean":"2023-10-13T23:56:14.759098+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-13T23:56:14.759098+0000","last_undegraded":"2023-10-13T23:56:14.759098+0000","last_fullsized":"2023-10-13T23:56:14.759098+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:00:17.635268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.761652+0000","last_change":"2023-10-13T23:56:14.757164+0000","last_active":"2023-10-13T23:56:14.761652+0000","last_peered":"2023-10-13T23:56:14.761652+0000","last_clean":"2023-10-13T23:56:14.761652+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-13T23:56:14.761652+0000","last_undegraded":"2023-10-13T23:56:14.761652+0000","last_fullsized":"2023-10-13T23:56:14.761652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:30:38.094808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.757356+0000","last_change":"2023-10-13T23:56:14.755160+0000","last_active":"2023-10-13T23:56:14.757356+0000","last_peered":"2023-10-13T23:56:14.757356+0000","last_clean":"2023-10-13T23:56:14.757356+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-13T23:56:14.757356+0000","last_undegraded":"2023-10-13T23:56:14.757356+0000","last_fullsized":"2023-10-13T23:56:14.757356+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:50:15.986492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.756230+0000","last_change":"2023-10-13T23:56:14.753469+0000","last_active":"2023-10-13T23:56:14.756230+0000","last_peered":"2023-10-13T23:56:14.756230+0000","last_clean":"2023-10-13T23:56:14.756230+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-13T23:56:14.756230+0000","last_undegraded":"2023-10-13T23:56:14.756230+0000","last_fullsized":"2023-10-13T23:56:14.756230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:16:53.104574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-13T23:56:15.748585+0000","last_change":"2023-10-13T23:56:14.754720+0000","last_active":"2023-10-13T23:56:15.748585+0000","last_peered":"2023-10-13T23:56:15.748585+0000","last_clean":"2023-10-13T23:56:15.748585+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-13T23:56:15.748585+0000","last_undegraded":"2023-10-13T23:56:15.748585+0000","last_fullsized":"2023-10-13T23:56:15.748585+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:39:43.759783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-13T23:56:15.748586+0000","last_change":"2023-10-13T23:56:14.754793+0000","last_active":"2023-10-13T23:56:15.748586+0000","last_peered":"2023-10-13T23:56:15.748586+0000","last_clean":"2023-10-13T23:56:15.748586+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-13T23:56:15.748586+0000","last_undegraded":"2023-10-13T23:56:15.748586+0000","last_fullsized":"2023-10-13T23:56:15.748586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:22:24.150498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-13T23:56:14.756823+0000","last_change":"2023-10-13T23:56:14.754310+0000","last_active":"2023-10-13T23:56:14.756823+0000","last_peered":"2023-10-13T23:56:14.756823+0000","last_clean":"2023-10-13T23:56:14.756823+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-13T23:56:14.756823+0000","last_undegraded":"2023-10-13T23:56:14.756823+0000","last_fullsized":"2023-10-13T23:56:14.756823+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:12:50.830266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-13T23:56:13.747465+0000","last_change":"2023-10-13T23:56:09.871340+0000","last_active":"2023-10-13T23:56:13.747465+0000","last_peered":"2023-10-13T23:56:13.747465+0000","last_clean":"2023-10-13T23:56:13.747465+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-13T23:56:13.747465+0000","last_undegraded":"2023-10-13T23:56:13.747465+0000","last_fullsized":"2023-10-13T23:56:13.747465+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_clean_scrub_stamp":"2023-10-13T23:56:08.695661+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:39:25.194520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12496,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":624088,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6248,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-13T23:56:18.121 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-13T23:56:18.121 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-13T23:56:18.121 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-13T23:56:18.122 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-13T23:56:18.410 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-13T23:56:18.410 INFO:teuthology.orchestra.run.smithi138.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-13T23:56:18.429 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-13T23:56:18.430 INFO:teuthology.run_tasks:Running task install... 2023-10-13T23:56:18.439 DEBUG:teuthology.task.install:project ceph 2023-10-13T23:56:18.439 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}} 2023-10-13T23:56:18.439 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'} 2023-10-13T23:56:18.439 INFO:teuthology.task.install:Using flavor: default 2023-10-13T23:56:18.446 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-13T23:56:18.447 INFO:teuthology.task.install:extra packages: [] 2023-10-13T23:56:18.447 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-key list | grep Ceph 2023-10-13T23:56:18.497 INFO:teuthology.orchestra.run.smithi138.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-13T23:56:18.520 INFO:teuthology.orchestra.run.smithi138.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-13T23:56:18.521 INFO:teuthology.orchestra.run.smithi138.stdout:uid [ unknown] Ceph.com (release key) 2023-10-13T23:56:18.521 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-13T23:56:18.521 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-13T23:56:18.521 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-13T23:56:18.744 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default/ 2023-10-13T23:56:18.745 INFO:teuthology.task.install.deb:Package version is 18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:56:18.855 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:56:18.856 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-13T23:56:18.870 DEBUG:teuthology.orchestra.run.smithi138:> sudo apt-get update 2023-10-13T23:56:19.162 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:56:19.163 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-13T23:56:19.168 INFO:teuthology.orchestra.run.smithi138.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy InRelease 2023-10-13T23:56:19.204 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:4 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release 2023-10-13T23:56:19.235 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-13T23:56:19.242 INFO:teuthology.orchestra.run.smithi138.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-13T23:56:19.320 INFO:teuthology.orchestra.run.smithi138.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-13T23:56:20.339 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:56:20.354 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6724-g01fff6a7-1jammy cephadm=18.0.0-6724-g01fff6a7-1jammy ceph-mds=18.0.0-6724-g01fff6a7-1jammy ceph-mgr=18.0.0-6724-g01fff6a7-1jammy ceph-common=18.0.0-6724-g01fff6a7-1jammy ceph-fuse=18.0.0-6724-g01fff6a7-1jammy ceph-test=18.0.0-6724-g01fff6a7-1jammy ceph-volume=18.0.0-6724-g01fff6a7-1jammy radosgw=18.0.0-6724-g01fff6a7-1jammy python3-rados=18.0.0-6724-g01fff6a7-1jammy python3-rgw=18.0.0-6724-g01fff6a7-1jammy python3-cephfs=18.0.0-6724-g01fff6a7-1jammy python3-rbd=18.0.0-6724-g01fff6a7-1jammy libcephfs2=18.0.0-6724-g01fff6a7-1jammy libcephfs-dev=18.0.0-6724-g01fff6a7-1jammy librados2=18.0.0-6724-g01fff6a7-1jammy librbd1=18.0.0-6724-g01fff6a7-1jammy rbd-fuse=18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:56:20.417 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:56:20.603 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2023-10-13T23:56:20.604 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2023-10-13T23:56:20.749 INFO:teuthology.orchestra.run.smithi138.stdout:ceph is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.749 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-common is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.749 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-fuse is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.750 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-mds is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.750 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-mgr is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.750 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-test is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.750 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-volume is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.751 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.751 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-dev is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.751 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs2 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.751 INFO:teuthology.orchestra.run.smithi138.stdout:librados2 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.751 INFO:teuthology.orchestra.run.smithi138.stdout:librbd1 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.752 INFO:teuthology.orchestra.run.smithi138.stdout:python3-cephfs is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.752 INFO:teuthology.orchestra.run.smithi138.stdout:python3-rados is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.752 INFO:teuthology.orchestra.run.smithi138.stdout:python3-rbd is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.752 INFO:teuthology.orchestra.run.smithi138.stdout:python3-rgw is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.752 INFO:teuthology.orchestra.run.smithi138.stdout:radosgw is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.752 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-13T23:56:20.753 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:56:20.753 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:56:20.753 INFO:teuthology.orchestra.run.smithi138.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:56:20.753 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:56:20.753 INFO:teuthology.orchestra.run.smithi138.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-13T23:56:20.754 INFO:teuthology.orchestra.run.smithi138.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-13T23:56:20.754 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:56:20.784 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 0 newly installed, 0 to remove and 256 not upgraded. 2023-10-13T23:56:20.784 INFO:teuthology.orchestra.run.smithi138.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-13T23:56:20.785 DEBUG:teuthology.orchestra.run.smithi138:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-13T23:56:20.845 INFO:teuthology.orchestra.run.smithi138.stdout:Reading package lists... 2023-10-13T23:56:21.030 INFO:teuthology.orchestra.run.smithi138.stdout:Building dependency tree... 2023-10-13T23:56:21.031 INFO:teuthology.orchestra.run.smithi138.stdout:Reading state information... 2023-10-13T23:56:21.193 INFO:teuthology.orchestra.run.smithi138.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:56:21.194 INFO:teuthology.orchestra.run.smithi138.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-13T23:56:21.194 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-13T23:56:21.194 INFO:teuthology.orchestra.run.smithi138.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-13T23:56:21.195 INFO:teuthology.orchestra.run.smithi138.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-13T23:56:21.195 INFO:teuthology.orchestra.run.smithi138.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-13T23:56:21.195 INFO:teuthology.orchestra.run.smithi138.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:56:21.195 INFO:teuthology.orchestra.run.smithi138.stdout:The following additional packages will be installed: 2023-10-13T23:56:21.196 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemblk1 2023-10-13T23:56:21.197 INFO:teuthology.orchestra.run.smithi138.stdout:Suggested packages: 2023-10-13T23:56:21.197 INFO:teuthology.orchestra.run.smithi138.stdout: gnuplot gfio python-scipy 2023-10-13T23:56:21.222 INFO:teuthology.orchestra.run.smithi138.stdout:The following NEW packages will be installed: 2023-10-13T23:56:21.222 INFO:teuthology.orchestra.run.smithi138.stdout: fio libpmemblk1 2023-10-13T23:56:21.273 INFO:teuthology.orchestra.run.smithi138.stdout:0 upgraded, 2 newly installed, 0 to remove and 256 not upgraded. 2023-10-13T23:56:21.273 INFO:teuthology.orchestra.run.smithi138.stdout:Need to get 4,112 kB of archives. 2023-10-13T23:56:21.273 INFO:teuthology.orchestra.run.smithi138.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-13T23:56:21.273 INFO:teuthology.orchestra.run.smithi138.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-13T23:56:21.331 INFO:teuthology.orchestra.run.smithi138.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-13T23:56:21.793 INFO:teuthology.orchestra.run.smithi138.stdout:Fetched 4,112 kB in 0s (18.1 MB/s) 2023-10-13T23:56:21.949 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-13T23:56:22.050 INFO:teuthology.orchestra.run.smithi138.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172814 files and directories currently installed.) 2023-10-13T23:56:22.055 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-13T23:56:22.140 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-13T23:56:22.486 INFO:teuthology.orchestra.run.smithi138.stdout:Selecting previously unselected package fio. 2023-10-13T23:56:22.514 INFO:teuthology.orchestra.run.smithi138.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-13T23:56:22.552 INFO:teuthology.orchestra.run.smithi138.stdout:Unpacking fio (3.28-1) ... 2023-10-13T23:56:23.088 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-13T23:56:23.200 INFO:teuthology.orchestra.run.smithi138.stdout:Setting up fio (3.28-1) ... 2023-10-13T23:56:23.696 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-13T23:56:25.041 INFO:teuthology.orchestra.run.smithi138.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-13T23:56:25.493 INFO:teuthology.orchestra.run.smithi138.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-13T23:56:25.496 DEBUG:teuthology.parallel:result is None 2023-10-13T23:56:25.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-13T23:56:25.639 DEBUG:teuthology.orchestra.run.smithi138:> dpkg-query -W -f '${Version}' ceph 2023-10-13T23:56:25.658 INFO:teuthology.orchestra.run.smithi138.stdout:18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:56:25.658 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:56:25.659 INFO:teuthology.task.install:The correct ceph version 18.0.0-6724-g01fff6a7-1jammy is installed. 2023-10-13T23:56:25.659 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-13T23:56:25.659 INFO:teuthology.run_tasks:Running task workunit... 2023-10-13T23:56:25.673 INFO:tasks.workunit:Pulling workunits from ref 735f68d7bf91d643813a8420fc2b11441f4bd8c4 2023-10-13T23:56:25.674 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-13T23:56:25.674 DEBUG:teuthology.orchestra.run.smithi138:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-13T23:56:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-13T23:56:25.705 INFO:teuthology.orchestra.run.smithi138.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-13T23:56:25.705 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-13T23:56:25.756 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-13T23:56:25.757 DEBUG:teuthology.orchestra.run.smithi138:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-13T23:56:25.804 INFO:tasks.workunit:timeout=3h 2023-10-13T23:56:25.805 INFO:tasks.workunit:cleanup=True 2023-10-13T23:56:25.805 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/idryomov/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 735f68d7bf91d643813a8420fc2b11441f4bd8c4 2023-10-13T23:56:25.854 INFO:tasks.workunit.client.0.smithi138.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-13T23:57:05.687 INFO:tasks.workunit.client.0.smithi138.stderr:Note: switching to '735f68d7bf91d643813a8420fc2b11441f4bd8c4'. 2023-10-13T23:57:05.687 INFO:tasks.workunit.client.0.smithi138.stderr: 2023-10-13T23:57:05.687 INFO:tasks.workunit.client.0.smithi138.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-13T23:57:05.688 INFO:tasks.workunit.client.0.smithi138.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-13T23:57:05.688 INFO:tasks.workunit.client.0.smithi138.stderr:state without impacting any branches by switching back to a branch. 2023-10-13T23:57:05.688 INFO:tasks.workunit.client.0.smithi138.stderr: 2023-10-13T23:57:05.688 INFO:tasks.workunit.client.0.smithi138.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-13T23:57:05.688 INFO:tasks.workunit.client.0.smithi138.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-13T23:57:05.689 INFO:tasks.workunit.client.0.smithi138.stderr: 2023-10-13T23:57:05.689 INFO:tasks.workunit.client.0.smithi138.stderr: git switch -c 2023-10-13T23:57:05.689 INFO:tasks.workunit.client.0.smithi138.stderr: 2023-10-13T23:57:05.689 INFO:tasks.workunit.client.0.smithi138.stderr:Or undo this operation with: 2023-10-13T23:57:05.689 INFO:tasks.workunit.client.0.smithi138.stderr: 2023-10-13T23:57:05.690 INFO:tasks.workunit.client.0.smithi138.stderr: git switch - 2023-10-13T23:57:05.690 INFO:tasks.workunit.client.0.smithi138.stderr: 2023-10-13T23:57:05.690 INFO:tasks.workunit.client.0.smithi138.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-13T23:57:05.690 INFO:tasks.workunit.client.0.smithi138.stderr: 2023-10-13T23:57:05.690 INFO:tasks.workunit.client.0.smithi138.stderr:HEAD is now at 735f68d7bf9 Merge branch 'wip-62891' of https://github.com/ajarr/ceph into HEAD 2023-10-13T23:57:05.693 DEBUG:teuthology.orchestra.run.smithi138:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-13T23:57:05.740 INFO:tasks.workunit.client.0.smithi138.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-13T23:57:05.743 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-13T23:57:05.744 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-13T23:57:06.539 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-13T23:57:06.634 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-13T23:57:06.683 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-13T23:57:06.684 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-13T23:57:06.685 INFO:tasks.workunit.client.0.smithi138.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-13T23:57:06.734 INFO:tasks.workunit.client.0.smithi138.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-13T23:57:06.737 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2023-10-13T23:57:06.737 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-13T23:57:06.784 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-13T23:57:06.785 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-13T23:57:06.786 DEBUG:teuthology.orchestra.run.smithi138:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-13T23:57:06.836 INFO:tasks.workunit.client.0.smithi138.stderr:+ POOL=rbd 2023-10-13T23:57:06.836 INFO:tasks.workunit.client.0.smithi138.stderr:+ IMAGE_PREFIX=image 2023-10-13T23:57:06.836 INFO:tasks.workunit.client.0.smithi138.stderr:+ NUM_IMAGES=20 2023-10-13T23:57:06.836 INFO:tasks.workunit.client.0.smithi138.stderr:+ RUN_TIME=3600 2023-10-13T23:57:06.837 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror pool enable rbd image 2023-10-13T23:57:06.897 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-13T23:57:06.935 INFO:tasks.workunit.client.0.smithi138.stdout:b4e6f93f-486e-414f-94b8-1de83c3bd022 2023-10-13T23:57:06.938 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i = 1 )) 2023-10-13T23:57:06.939 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:06.939 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-13T23:57:08.690 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-13T23:57:09.700 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:09.714 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-13T23:57:09.777 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:09.777 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:09.777 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-13T23:57:09.826 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-13T23:57:10.709 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:10.722 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-13T23:57:10.785 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:10.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:10.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-13T23:57:10.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-13T23:57:11.720 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:11.734 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-13T23:57:11.795 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:11.795 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:11.795 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-13T23:57:11.845 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-13T23:57:12.722 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:12.738 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-13T23:57:12.799 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:12.799 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:12.800 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-13T23:57:12.852 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-13T23:57:13.732 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:13.746 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-13T23:57:13.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:13.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:13.808 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-13T23:57:13.859 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-13T23:57:14.753 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:14.768 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-13T23:57:14.829 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:14.829 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:14.829 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-13T23:57:14.878 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-13T23:57:15.757 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:15.769 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-13T23:57:15.827 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:15.827 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:15.828 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-13T23:57:15.874 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-13T23:57:16.757 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:16.771 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-13T23:57:16.832 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:16.832 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:16.833 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-13T23:57:16.879 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-13T23:57:17.644 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:17.656 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-13T23:57:17.718 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:17.718 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:17.718 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-13T23:57:17.767 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-13T23:57:18.774 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:18.790 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-13T23:57:18.849 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:18.849 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:18.850 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-13T23:57:18.899 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-13T23:57:19.780 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:19.795 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-13T23:57:19.857 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:19.857 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:19.858 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-13T23:57:19.907 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-13T23:57:20.793 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:20.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-13T23:57:20.870 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:20.870 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:20.871 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-13T23:57:20.919 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-13T23:57:21.800 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:21.814 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-13T23:57:21.874 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:21.874 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:21.874 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-13T23:57:21.922 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-13T23:57:22.801 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:22.814 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-13T23:57:22.877 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:22.878 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:22.878 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-13T23:57:22.926 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-13T23:57:23.815 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:23.826 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-13T23:57:23.887 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:23.887 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:23.887 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-13T23:57:23.939 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-13T23:57:24.809 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:24.822 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-13T23:57:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-13T23:57:24.931 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-13T23:57:25.827 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:25.839 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-13T23:57:25.901 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:25.901 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:25.902 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-13T23:57:25.951 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-13T23:57:26.825 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:26.837 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-13T23:57:26.897 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:26.898 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:26.898 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-13T23:57:26.945 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-13T23:57:27.838 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:27.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-13T23:57:27.908 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:27.909 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:27.909 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-13T23:57:27.959 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-13T23:57:28.840 INFO:tasks.workunit.client.0.smithi138.stdout:Mirroring enabled 2023-10-13T23:57:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-13T23:57:28.915 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:28.916 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:28.916 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i = 1 )) 2023-10-13T23:57:28.917 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:28.917 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image1 2023-10-13T23:57:29.118 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-13T23:57:29.118 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.118 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.118 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.119 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image2 2023-10-13T23:57:29.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-13T23:57:29.191 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.191 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.191 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.192 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image3 2023-10-13T23:57:29.262 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-13T23:57:29.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.263 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image4 2023-10-13T23:57:29.335 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-13T23:57:29.336 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.336 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.337 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.337 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image5 2023-10-13T23:57:29.444 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-13T23:57:29.444 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.445 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.445 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.445 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image6 2023-10-13T23:57:29.544 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-13T23:57:29.545 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.545 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.545 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.545 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image7 2023-10-13T23:57:29.662 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-13T23:57:29.663 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.663 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.663 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.664 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image8 2023-10-13T23:57:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-13T23:57:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.803 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image9 2023-10-13T23:57:29.923 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-13T23:57:29.923 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:29.924 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:29.924 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:29.924 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image10 2023-10-13T23:57:30.034 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-13T23:57:30.035 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:30.035 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:30.035 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:30.035 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image11 2023-10-13T23:57:30.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:30.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:30.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:30.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:30.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:30.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:30.187 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-13T23:57:30.187 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:30.188 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:30.188 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:30.188 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image12 2023-10-13T23:57:30.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-13T23:57:30.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:30.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:30.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:30.335 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image13 2023-10-13T23:57:30.519 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-13T23:57:30.519 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:30.519 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:30.520 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image14 2023-10-13T23:57:30.522 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:30.704 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-13T23:57:30.704 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:30.705 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:30.705 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image15 2023-10-13T23:57:30.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:30.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:30.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:30.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-13T23:57:30.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:30.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:30.845 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image16 2023-10-13T23:57:30.847 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:31.097 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-13T23:57:31.098 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:31.098 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:31.098 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:31.099 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image17 2023-10-13T23:57:31.271 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-13T23:57:31.272 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:31.272 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:31.272 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:31.274 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image18 2023-10-13T23:57:31.484 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-13T23:57:31.484 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:31.484 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:31.484 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:31.485 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image19 2023-10-13T23:57:31.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:31.693 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-13T23:57:31.693 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:31.694 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:31.694 INFO:tasks.workunit.client.0.smithi138.stderr:++ sudo rbd device map rbd/image20 2023-10-13T23:57:31.715 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:31.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:31.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:31.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:31.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:31.900 INFO:tasks.workunit.client.0.smithi138.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-13T23:57:31.901 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-13T23:57:31.901 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 20 )) 2023-10-13T23:57:31.901 INFO:tasks.workunit.client.0.smithi138.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-13T23:57:31.902 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:57:31.902 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241451 2023-10-13T23:57:31.903 INFO:tasks.workunit.client.0.smithi138.stderr:+ END_TIME=1697245051 2023-10-13T23:57:31.903 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR= 2023-10-13T23:57:31.903 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR= 2023-10-13T23:57:31.904 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:57:31.904 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n '' ]] 2023-10-13T23:57:31.904 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:57:31.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:31.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:31.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:31.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:31.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:31.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:32.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:32.638+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:33.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:57:33.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:33.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:33.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:33.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:33.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:33.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:33.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:33.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:33.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:33.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:33.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:33.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:33.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:33.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:33.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:33.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:33.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:33.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:33.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:33.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:33.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:33.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:33.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:33.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:33.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:33.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:33.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:33.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:34.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:34.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:34.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:34.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:34.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:34.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:35.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:35.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:35.290 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:35.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:35.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:35.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:35.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:35.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:35.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:35.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:35.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:35.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:35.789 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:35.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:35.791 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:35.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:35.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:35.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:35.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:35.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:35.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:35.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:36.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:36.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:36.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:36.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:36.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:36.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:36.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:36.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:37.217 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:37.213+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:37.228 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:37.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:37.229 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:37.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:37.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:37.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:37.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:37.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:37.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:37.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:37.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-13T23:57:37.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:57:37.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:57:37.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:37.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:37.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:37.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:37.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:37.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:37.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:37.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:38.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:38.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:38.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:38.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:38.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.713+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:38.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.713+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:38.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:38.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:38.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:38.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:38.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:38.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.817+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:38.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.817+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:38.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.817+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:38.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.829+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:38.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:38.829+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:39.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:57:39.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:39.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:39.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:39.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:39.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:57:39.961 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:39.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.957+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:39.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.957+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:39.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:39.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:39.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:39.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:40.578 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:40.580 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:40.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:40.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:40.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:57:40.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:40.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:40.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:40.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:40.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:40.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:40.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:57:40.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:40.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:40.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:40.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:40.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:40.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:40.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:40.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:40.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:40.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:40.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:40.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:40.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:40.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:40.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:40.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:40.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:40.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:41.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:57:41.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:41.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:41.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:41.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:41.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:41.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:41.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:41.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:41.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:41.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:41.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:41.901 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:57:41.902 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:57:41.903 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:57:41.910 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:57:42.111 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-13T23:57:42.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:42.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.129+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:42.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.133+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:42.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.133+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:42.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.133+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:42.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:42.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:42.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:42.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:42.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:42.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:42.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:42.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:42.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:42.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-13T23:57:42.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:57:42.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:42.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:42.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:42.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:42.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:42.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:42.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:42.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.493+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:57:42.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.493+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:42.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:42.493+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:42.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:42.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:42.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:42.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:42.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:43.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:43.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:43.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:43.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:43.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:43.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:43.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:43.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:43.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:43.548 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2213336685 2023-10-13T23:57:43.548 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:57:43.550 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241463 2023-10-13T23:57:43.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:57:43.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2213336685 ]] 2023-10-13T23:57:43.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2213336685 != '' ]] 2023-10-13T23:57:43.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2213336685 2023-10-13T23:57:43.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:43.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:43.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:43.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:43.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:44.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:44.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:44.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:44.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:44.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:44.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:44.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:44.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:44.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:44.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:44.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:44.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:45.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:45.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:45.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:45.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:45.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:45.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:45.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:45.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:45.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:45.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:45.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:45.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:45.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:45.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:45.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:45.277+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:45.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:45.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:45.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:45.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:45.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:45.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:45.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:45.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:45.627 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2213336685 until 2023-10-14T00:57:44.957324+0000 (3600 sec) 2023-10-13T23:57:45.669 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-13T23:57:45.670 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2213336685 2023-10-13T23:57:46.790 INFO:tasks.workunit.client.0.smithi138.stderr:listed 1 entries 2023-10-13T23:57:46.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2213336685 2023-10-13T23:57:46.835 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:57:47.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:47.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:47.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:47.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:47.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:47.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:47.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:47.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:48.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:48.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:48.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:48.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:48.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:48.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:48.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:48.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:48.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:48.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:48.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:48.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:48.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:48.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:48.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:48.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:48.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:48.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:48.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:48.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:49.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.529+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-13T23:57:49.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:57:49.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:49.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:49.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:49.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:49.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:49.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:49.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:57:49.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:49.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:49.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:50.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:50.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:50.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:50.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:50.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:50.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:50.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:50.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:50.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:50.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:50.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:50.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:51.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:57:51.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:51.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.469+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:51.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.469+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:51.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:51.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:51.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:51.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:57:51.677 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.677+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:57:51.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.677+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:51.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:57:51.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:57:51.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:51.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:57:52.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:52.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:52.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:52.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:52.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:52.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:52.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:57:52.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:52.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:52.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:52.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:52.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:57:52.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:57:53.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:53.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:53.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:53.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:53.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:53.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:53.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:53.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:53.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:53.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:53.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:53.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:53.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:53.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:54.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:54.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:54.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:54.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:54.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:54.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:54.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:54.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:54.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:54.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:54.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:54.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:54.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:55.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:55.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:55.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:55.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:55.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:55.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:55.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:55.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:55.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:55.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:56.838 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:57:56.839 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:57:56.839 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:57:56.844 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:57:58.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:58.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:57:58.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:58.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:58.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:58.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:58.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:58.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:57:58.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:58.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:57:58.288 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2213336685 2023-10-13T23:57:58.288 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:57:58.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241478 2023-10-13T23:57:58.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:57:58.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2213336685 ]] 2023-10-13T23:57:58.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2213336685 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\2\1\3\3\3\6\6\8\5 ]] 2023-10-13T23:57:58.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:57:58.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:58.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:58.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:58.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:57:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:59.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:57:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:59.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:57:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:59.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:57:59.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:57:59.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:00.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:00.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:00.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:00.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:00.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:00.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:00.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:00.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:00.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:01.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:01.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:01.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:01.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:01.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:01.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:58:01.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:01.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:01.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:01.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:01.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:01.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:01.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:01.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.593+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:01.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:01.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:01.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:01.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:01.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:01.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:01.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:02.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:02.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:02.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:02.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:02.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:02.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:02.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:02.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:02.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:02.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:02.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:02.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:03.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:03.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:03.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:03.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:03.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:03.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:03.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:03.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:03.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:06.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:58:06.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:06.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:06.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:06.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:06.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:06.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:06.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:06.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:06.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:06.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:06.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:06.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:06.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:06.901+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:06.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:06.901+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:06.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:06.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:06.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:06.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:06.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:06.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:07.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:07.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:07.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:07.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:07.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:07.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:07.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:07.408+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:07.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:07.408+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:07.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:07.408+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:07.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:07.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:07.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:07.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:07.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:07.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:07.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:07.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:07.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:07.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:08.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:08.292 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:58:08.293 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:58:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:58:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:58:08.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:08.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:08.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:08.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:08.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:08.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:08.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:08.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:08.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:08.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:08.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:08.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:08.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:08.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:08.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:09.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:09.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:09.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:09.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:09.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:09.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:09.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:09.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:09.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:09.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:09.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:09.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:09.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:09.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:09.589 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2213336685 2023-10-13T23:58:09.589 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:58:09.590 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241489 2023-10-13T23:58:09.590 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:58:09.591 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2213336685 ]] 2023-10-13T23:58:09.591 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2213336685 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\2\1\3\3\3\6\6\8\5 ]] 2023-10-13T23:58:09.591 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:58:10.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:10.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:10.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:10.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:10.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:10.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:10.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:10.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:10.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:10.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:10.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:10.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:10.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:10.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:10.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:10.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:10.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:10.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:10.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:10.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:11.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:11.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:11.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:11.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:11.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:11.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:11.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:11.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:11.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:11.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:11.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:11.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:11.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:11.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:11.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:11.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:11.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:11.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:11.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:12.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:12.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:12.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:12.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:12.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:12.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:12.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:12.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:12.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:13.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:13.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:13.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:13.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:13.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:13.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:13.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:13.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:13.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:13.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:13.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:14.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:14.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:14.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:14.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:14.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:14.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:14.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:14.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:14.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:14.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:15.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:15.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:15.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:15.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:15.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:15.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:15.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:15.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:15.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:15.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:15.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:15.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:15.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:15.640+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:15.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:15.640+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:15.643 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:15.640+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:15.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:15.656+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:15.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:15.660+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:15.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:15.660+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:15.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:15.660+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:16.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:16.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.456+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:16.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:16.469 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:16.469 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:16.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:16.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:16.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:16.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:16.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:16.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.824+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:16.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:16.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:16.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:16.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:16.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:16.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:16.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:16.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:17.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:17.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:17.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:17.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:17.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:17.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:18.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:18.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:18.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:18.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:18.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:18.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:18.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:18.800+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:18.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:18.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:18.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:18.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:18.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:18.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:19.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:19.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:19.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:19.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:19.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:19.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:19.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:19.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:19.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:19.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:19.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:19.448+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:19.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:19.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:19.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:19.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:19.616 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:58:19.617 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:58:19.617 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:58:19.638 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:58:19.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:19.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:19.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:19.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:19.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:19.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:19.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:20.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:20.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:20.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:20.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:20.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:20.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:20.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:20.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:20.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:20.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:20.532+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:20.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:20.532+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:20.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:20.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:20.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:20.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:21.229 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1028345447 2023-10-13T23:58:21.229 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:58:21.232 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241501 2023-10-13T23:58:21.232 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:58:21.233 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1028345447 ]] 2023-10-13T23:58:21.233 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1028345447 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\2\1\3\3\3\6\6\8\5 ]] 2023-10-13T23:58:21.233 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/1028345447 2023-10-13T23:58:21.313 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.312+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:21.314 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.312+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:21.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:21.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:21.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:21.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:21.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:21.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:21.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:22.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:22.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:22.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:22.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:22.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:22.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:22.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:22.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:22.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:22.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:22.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:22.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:22.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:22.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:22.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:22.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:22.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:22.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:22.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:22.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:22.992 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:22.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:22.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:22.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:23.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:23.014 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:23.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:23.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:23.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:23.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.460+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-13T23:58:23.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.484+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:23.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:23.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:23.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:23.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:23.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:23.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:23.633 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/1028345447 until 2023-10-14T00:58:22.994645+0000 (3600 sec) 2023-10-13T23:58:23.686 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-13T23:58:23.705 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/1028345447 2023-10-13T23:58:24.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:24.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:24.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:24.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:24.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:24.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:24.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:24.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:24.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:24.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:24.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:24.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:24.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:24.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:24.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:24.892 INFO:tasks.workunit.client.0.smithi138.stderr:listed 2 entries 2023-10-13T23:58:24.942 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/1028345447 2023-10-13T23:58:24.942 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:58:25.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:25.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:25.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:25.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:25.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:25.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:25.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:25.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:25.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:25.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:25.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:25.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:25.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:25.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:25.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:25.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:25.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:29.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:29.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:29.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:29.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:29.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:29.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:29.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:29.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:29.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:29.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:29.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:29.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:29.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:30.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:30.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:30.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:30.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:30.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:30.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:30.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:30.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:30.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:31.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:31.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:31.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:31.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:31.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:31.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:31.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:31.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:31.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:31.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:33.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.132+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:33.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.132+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:33.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.132+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:33.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-13T23:58:33.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:33.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:33.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:33.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:33.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:34.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:34.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:34.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:34.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:34.097 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:34.096+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:34.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:34.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:34.102 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:34.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:34.945 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:58:34.946 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:58:34.949 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:58:34.950 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:58:35.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:35.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:35.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:35.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:35.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:35.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:35.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:35.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:35.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:36.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:35.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:36.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:36.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:36.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:36.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:36.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:36.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:36.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:36.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:36.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:36.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:36.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:36.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:36.293 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1028345447 2023-10-13T23:58:36.294 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:58:36.294 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241516 2023-10-13T23:58:36.295 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:58:36.295 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1028345447 ]] 2023-10-13T23:58:36.295 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1028345447 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\2\8\3\4\5\4\4\7 ]] 2023-10-13T23:58:36.295 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:58:37.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:37.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:37.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:37.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:37.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:37.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:37.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:37.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:38.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:38.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:38.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:38.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:38.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:38.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:38.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:38.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:38.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:38.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:41.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:41.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:41.141 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:41.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:41.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:41.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:41.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:41.451+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:41.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:41.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:41.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:41.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:41.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:41.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:41.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:41.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:41.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:41.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:41.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:41.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:41.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:41.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:41.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:41.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:44.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:44.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:44.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:44.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:44.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:44.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:44.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:58:44.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:44.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:44.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:44.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:44.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:44.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:44.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:45.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:45.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:45.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:45.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:45.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:45.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:45.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:45.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:45.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:45.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:45.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:45.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:45.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:45.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:45.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:45.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:45.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:45.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:45.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:45.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:45.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:45.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:45.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:45.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:46.301 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:58:46.304 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:58:46.311 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:58:46.326 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:58:47.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:47.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:47.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:47.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:47.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:47.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:47.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:47.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:47.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:47.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:47.761 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1028345447 2023-10-13T23:58:47.762 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:58:47.763 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241527 2023-10-13T23:58:47.763 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:58:47.763 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1028345447 ]] 2023-10-13T23:58:47.763 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1028345447 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\2\8\3\4\5\4\4\7 ]] 2023-10-13T23:58:47.763 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:58:48.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:48.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:48.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:48.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:48.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:48.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:48.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:48.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:48.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:48.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:48.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:48.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:48.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:48.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:49.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:49.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:49.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:49.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:49.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:49.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:49.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:49.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:49.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:49.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:49.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:49.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:50.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:50.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:50.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:50.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:50.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:50.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:50.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:50.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:50.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:50.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:50.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:50.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:50.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:50.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:51.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:51.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:51.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:51.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:51.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:51.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:51.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:51.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:51.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:51.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:51.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:51.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:51.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:51.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:52.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:52.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:52.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:52.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:52.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:52.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:53.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:53.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:53.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:53.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:53.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:53.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:53.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:53.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:54.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:54.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:54.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:54.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:54.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:54.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:54.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:54.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:54.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:58:54.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:54.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:54.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:54.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:54.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:54.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:58:54.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:54.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:54.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:54.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:54.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:54.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:54.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:54.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:54.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:54.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:54.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:54.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:54.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:54.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:54.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:54.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:54.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:54.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:54.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:58:54.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:54.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:54.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:54.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:56.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:56.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:58:56.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:56.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:58:56.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:56.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:58:57.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:56.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:58:57.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:58:56.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:58:57.766 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:58:57.767 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:58:57.767 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:58:57.789 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:58:59.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:58:59.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:59.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:59.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:59.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:59.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:59.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:58:59.241 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1028345447 2023-10-13T23:58:59.242 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:58:59.243 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241539 2023-10-13T23:58:59.243 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:58:59.243 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1028345447 ]] 2023-10-13T23:58:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1028345447 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\2\8\3\4\5\4\4\7 ]] 2023-10-13T23:58:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:58:59.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:58:59.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:58:59.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:58:59.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:58:59.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:58:59.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:58:59.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.007+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.007+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d86c9d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.011+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.011+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d86c9f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.015+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.015+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8734100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.025 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.019+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.025 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.019+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8734300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.027+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.027+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8734500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.035 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.031+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.035 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.031+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d86c9c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.040 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.035+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.040 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.035+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8465e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.043 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.039+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.043 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.039+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d7ab1a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.099+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.099+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8734600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.107+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.107+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8734800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.107+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.107+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8734300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.126 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.123+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.126 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.123+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.126 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.123+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8734080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.126 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.123+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d86c9d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.131+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.131+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8465c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.135+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.135+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d86c9f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.138 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.135+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.138 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.135+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d83c5a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.141 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.139+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.141 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.139+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d870e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.153 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.151+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.153 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.151+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d870e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.162 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.159+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.163 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.159+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d870e480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.166 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.163+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.166 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-13T23:59:00.163+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d870e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-13T23:59:00.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:00.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:00.521 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:00.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:00.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:00.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:00.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:00.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:00.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:00.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:00.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:00.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:00.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:00.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:02.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:02.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:02.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:02.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:02.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:02.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:02.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:02.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:02.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:02.543+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:02.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:02.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:02.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:02.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:02.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:02.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:03.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:03.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:03.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:03.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:03.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:03.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:03.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:03.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:03.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:03.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:03.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:04.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:04.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:04.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:04.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:04.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:04.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:05.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:05.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:05.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:05.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:05.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:05.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:05.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:05.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:05.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:05.950+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:59:05.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:05.950+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:05.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:05.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:06.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:06.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:06.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:06.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:06.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:06.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:06.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:06.450+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:06.469 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:06.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:08.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:08.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:08.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:08.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:08.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:08.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:08.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:08.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:09.249 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:59:09.258 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:59:09.264 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:59:09.266 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:59:09.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:09.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:09.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:09.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:09.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:09.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:09.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:09.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:09.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:09.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:09.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:09.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:09.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:09.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:10.732 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1028345447 2023-10-13T23:59:10.741 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:59:10.749 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241550 2023-10-13T23:59:10.749 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:59:10.749 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1028345447 ]] 2023-10-13T23:59:10.750 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1028345447 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\2\8\3\4\5\4\4\7 ]] 2023-10-13T23:59:10.750 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:59:11.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:11.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:11.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:11.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:11.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:11.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:11.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:11.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:11.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:11.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:11.712 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:11.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:11.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:11.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:11.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:11.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:11.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:11.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:11.741 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:11.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:14.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:14.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:14.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:14.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:14.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:14.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:14.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:14.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:14.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:14.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:15.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:15.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:15.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:15.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:15.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:15.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:15.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:15.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:15.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:15.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:15.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:15.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:16.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:16.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:16.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:16.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:16.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:16.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:16.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:16.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:16.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:59:16.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:16.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:16.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:16.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:16.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:16.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:16.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:17.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:17.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:17.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:17.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:18.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:17.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:18.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:18.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:18.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:18.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:18.081 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:18.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:18.096 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:18.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:18.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:18.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:18.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:18.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:18.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:18.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:18.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:18.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:18.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:18.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:18.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:18.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:18.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:18.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:18.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:18.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:18.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:18.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:20.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:20.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:59:20.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:20.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:20.626 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:20.622+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:20.627 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:20.622+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:20.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:20.630+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:20.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:20.630+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:20.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:20.630+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:20.764 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:59:20.764 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:59:20.777 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:59:20.797 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:59:21.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:59:21.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:21.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:21.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:21.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:21.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:21.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:21.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:59:21.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:21.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:21.568 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:21.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:21.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:21.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:21.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:21.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:21.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:21.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:21.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:21.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:21.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:21.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:21.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:21.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:21.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:22.386 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1028345447 2023-10-13T23:59:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:59:22.394 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241562 2023-10-13T23:59:22.394 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:59:22.394 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1028345447 ]] 2023-10-13T23:59:22.395 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1028345447 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\2\8\3\4\5\4\4\7 ]] 2023-10-13T23:59:22.395 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:59:24.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:24.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:24.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:24.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:24.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:24.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:24.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:24.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:24.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:24.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:24.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:24.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:24.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:25.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:25.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:25.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:25.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:26.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:26.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:26.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:26.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:26.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:26.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:26.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:26.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:26.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:27.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:27.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:27.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:27.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:27.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:27.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:27.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:27.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:27.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:30.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:30.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:30.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:30.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:30.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:30.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:30.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:30.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:30.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:30.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:30.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:30.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:30.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:30.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:30.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:30.950+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:30.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:30.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:30.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:30.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:30.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:30.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:31.466 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:31.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:31.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:31.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:31.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:31.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:31.971 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.966+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:31.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:31.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:31.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:31.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:32.403 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:59:32.404 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:59:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:59:32.434 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:59:33.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/543244662 2023-10-13T23:59:33.812 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:59:33.813 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241573 2023-10-13T23:59:33.814 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:59:33.814 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/543244662 ]] 2023-10-13T23:59:33.814 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/543244662 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\2\8\3\4\5\4\4\7 ]] 2023-10-13T23:59:33.814 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/543244662 2023-10-13T23:59:34.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:34.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:34.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:34.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:34.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:34.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:34.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:34.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:34.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:35.728 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/543244662 until 2023-10-14T00:59:35.015498+0000 (3600 sec) 2023-10-13T23:59:35.775 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-13T23:59:35.775 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/543244662 2023-10-13T23:59:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:listed 3 entries 2023-10-13T23:59:36.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:36.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:36.931 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:36.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:36.938 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/543244662 2023-10-13T23:59:36.938 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:59:36.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:36.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:36.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:36.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:36.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:36.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:36.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:36.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:37.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:37.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:37.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:37.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:37.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:37.494+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:37.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:37.518+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:37.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:37.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:37.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:37.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:37.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:37.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:39.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:39.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:39.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:39.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:39.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:39.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:39.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:39.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:39.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:39.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:39.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:39.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:39.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:39.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:39.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:39.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:40.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:40.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:40.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:40.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:40.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:40.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:40.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:40.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:40.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:40.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:40.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:40.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:41.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:41.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:41.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:41.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:41.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:41.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:41.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:41.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:41.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:41.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:41.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:41.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:59:41.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:41.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:41.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:41.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:41.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:43.907 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:43.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:43.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:43.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:43.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:43.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:43.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:43.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:43.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:43.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:44.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:44.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:44.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:44.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:44.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:44.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:44.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:44.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:44.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:44.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:44.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:44.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:44.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:45.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-13T23:59:45.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-13T23:59:45.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:45.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:45.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:45.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:45.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:45.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:45.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:45.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:46.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:46.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:46.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:46.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:46.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:46.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:46.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:46.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:46.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:46.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:46.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:46.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:46.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:46.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:46.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:46.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:46.957 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:59:46.960 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:59:46.961 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:59:46.963 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:59:47.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:46.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:47.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:47.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:47.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:47.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:47.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:47.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:47.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:47.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:47.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:47.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:47.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:47.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:47.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:47.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:48.357 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/543244662 2023-10-13T23:59:48.358 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:59:48.359 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241588 2023-10-13T23:59:48.359 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:59:48.359 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/543244662 ]] 2023-10-13T23:59:48.359 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/543244662 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\5\4\3\2\4\4\6\6\2 ]] 2023-10-13T23:59:48.359 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-13T23:59:49.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:49.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:49.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:49.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:49.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:49.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:49.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:49.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:49.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:49.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:49.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:49.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:49.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:49.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:59:49.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:49.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:49.766 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:49.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:49.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:49.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:49.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:49.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:49.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:49.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:49.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:49.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:51.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:51.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:59:51.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:51.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:51.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:51.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:51.111 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:51.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:51.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:51.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:51.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:51.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:51.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:51.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:51.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:51.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:51.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:51.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:51.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:51.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:51.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:51.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:51.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:51.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:52.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:52.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:52.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:52.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:52.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:52.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:52.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:52.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:52.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:52.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:52.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:52.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:53.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:53.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:53.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:53.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:53.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:53.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:53.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:53.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:53.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:53.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:59:53.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:53.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:53.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:53.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:54.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:54.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:54.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:54.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:54.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:54.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:54.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:54.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:54.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:54.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:54.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:54.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:54.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:54.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:54.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-13T23:59:54.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:54.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:54.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:54.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:54.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:54.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:54.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:54.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:54.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:54.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:54.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:54.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:55.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:55.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:55.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:55.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:55.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:55.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:55.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:55.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:57.643 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:57.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:57.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:57.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-13T23:59:57.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:57.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-13T23:59:57.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:57.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-13T23:59:57.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:57.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-13T23:59:57.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:57.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:57.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-13T23:59:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:57.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-13T23:59:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:57.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-13T23:59:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:57.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:57.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:57.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:57.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:57.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:58.360 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-13T23:59:58.361 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-13T23:59:58.368 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-13T23:59:58.385 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-13T23:59:58.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:58.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-13T23:59:59.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:59.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:59.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:59.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-13T23:59:59.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:59.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-13T23:59:59.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:59.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-13T23:59:59.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-13T23:59:59.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-13T23:59:59.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-13T23:59:59.925+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-13T23:59:59.965 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/543244662 2023-10-13T23:59:59.966 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-13T23:59:59.966 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241599 2023-10-13T23:59:59.966 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-13T23:59:59.967 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/543244662 ]] 2023-10-13T23:59:59.967 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/543244662 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\5\4\3\2\4\4\6\6\2 ]] 2023-10-13T23:59:59.967 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:00:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.005+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.005+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.005+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.005+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.009+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.009+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.027 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.021+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.021+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.033+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.033+0000 7f082630d640 -1 librbd::ImageState: 0x5627d83c4480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.037+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.043 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.037+0000 7f082630d640 -1 librbd::ImageState: 0x5627d870e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.065+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.065+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.065+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.065+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.065+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.069+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8465c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.069+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.069+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.085+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.085+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.089+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.089+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8465a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.177 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.177 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.177 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.178 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.178 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.178 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.178 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.178 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d870fd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae6400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d83c5500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.180 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.180 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cfb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.180 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:00:00.169+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:00:00.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:00.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:00.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:00.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:00.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:00.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:00.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:00.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:00.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:00.561+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:00.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:00.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:00.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:01.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:00.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:01.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:01.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:00:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:01.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:01.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:01.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:01.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:01.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:01.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:01.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:02.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:02.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:02.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:02.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:02.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:02.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:02.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:02.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:02.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:03.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:03.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:03.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:03.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:03.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:03.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:03.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:03.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:03.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:03.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:03.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:03.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:03.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:03.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:03.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:03.925+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:04.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:04.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:04.203 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:04.197+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:04.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:04.197+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:04.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:04.209+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:04.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:04.209+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:05.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:05.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:05.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:05.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:05.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:05.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:05.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:05.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:06.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:06.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:06.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:06.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:07.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:07.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:07.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:07.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:07.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:07.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:07.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:07.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:07.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:07.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:07.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:07.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:08.531 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:08.525+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:08.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:08.525+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:08.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:08.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:08.559 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:08.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:08.559 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:08.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:09.968 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:00:09.968 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:00:09.970 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:00:09.971 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:00:10.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:10.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:10.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:10.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:10.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:10.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:10.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:10.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:10.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:10.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:10.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:10.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:10.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:10.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:11.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/543244662 2023-10-14T00:00:11.443 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:00:11.443 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241611 2023-10-14T00:00:11.443 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:00:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/543244662 ]] 2023-10-14T00:00:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/543244662 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\5\4\3\2\4\4\6\6\2 ]] 2023-10-14T00:00:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:00:11.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:11.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:11.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:11.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:11.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:11.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:12.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:12.325+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:12.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:12.325+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:12.348 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:12.341+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:12.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:12.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:12.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:12.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:12.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:12.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:12.376 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:12.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:12.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:12.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:12.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:12.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:12.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:12.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:12.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:13.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:12.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:13.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:13.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:13.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:13.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:13.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:13.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:13.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:13.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:13.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:13.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:13.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:13.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:13.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:14.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:14.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:14.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:14.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:14.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:14.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:14.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:14.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:14.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:15.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:15.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:15.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:15.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:15.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:00:15.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:15.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:15.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:15.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:15.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:15.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:17.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:17.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:17.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:17.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:17.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:17.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:17.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:17.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:17.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:17.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:18.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:18.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:18.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:18.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:18.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:18.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:18.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:18.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:18.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:18.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:18.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:18.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:18.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:18.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:19.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:19.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:19.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:19.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:19.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:19.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:19.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:19.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:19.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:19.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:19.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:19.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:19.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:19.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:19.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:19.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:19.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:19.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:19.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:20.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:20.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:20.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:20.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:20.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:20.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:20.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:20.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:20.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:21.022 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:21.021+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:21.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:21.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:21.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:21.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:21.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:21.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:21.449 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:00:21.449 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:00:21.450 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:00:21.457 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:00:21.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:21.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:21.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:21.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:21.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:21.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:21.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:21.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:21.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:21.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:21.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:21.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:21.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:21.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:22.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:22.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:22.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:22.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:22.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:22.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:22.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:22.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:22.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:22.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:22.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/543244662 2023-10-14T00:00:22.805 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:00:22.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241622 2023-10-14T00:00:22.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:00:22.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/543244662 ]] 2023-10-14T00:00:22.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/543244662 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\5\4\3\2\4\4\6\6\2 ]] 2023-10-14T00:00:22.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:00:23.118 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:23.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:23.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:23.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:23.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:23.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:23.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:23.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:23.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:23.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:23.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:23.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:23.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:23.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:23.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:23.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:23.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:23.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:23.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:23.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:24.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.709+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:00:24.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:24.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:24.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.745+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:24.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:24.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:24.785 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:24.785 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:24.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:25.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:25.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:25.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:25.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:25.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:25.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:25.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:25.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:25.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:25.961+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:25.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:25.961+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:25.984 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:25.981+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:26.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:26.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:26.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:26.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:26.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:26.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:26.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:26.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:29.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:29.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:29.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:29.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:29.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:29.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:29.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:29.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:29.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:29.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:29.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:29.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:30.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:30.745+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:30.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:30.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:30.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:30.753+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:30.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:30.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:30.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:30.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:30.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:30.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:31.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:31.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:31.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:31.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:31.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:31.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:31.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:31.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:31.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:31.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:31.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:31.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:31.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:31.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:31.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:31.428 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:31.428 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:31.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:31.487 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:31.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:31.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:31.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:31.706 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:31.713 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.713+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:31.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.713+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:31.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:31.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:31.719 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:31.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:32.823 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:00:32.824 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:00:32.826 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:00:32.837 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:00:33.166 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:00:33.165+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:00:34.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:34.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:34.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:34.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:34.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:34.230 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/543244662 2023-10-14T00:00:34.230 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:00:34.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241634 2023-10-14T00:00:34.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:00:34.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/543244662 ]] 2023-10-14T00:00:34.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/543244662 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\5\4\3\2\4\4\6\6\2 ]] 2023-10-14T00:00:34.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:00:34.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:34.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:34.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:34.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.593+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:34.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:34.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:34.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:34.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:34.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:00:34.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:34.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:34.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:34.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:34.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:34.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:34.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:34.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:34.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:34.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:34.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:35.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:35.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:35.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:35.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:35.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:35.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:35.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:35.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:35.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:36.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:36.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:36.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:36.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:36.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:36.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:36.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:36.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:36.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:36.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:37.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:37.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:37.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:37.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:37.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:37.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:37.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:37.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:37.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:37.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:38.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:38.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:38.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:38.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:38.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:38.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:38.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:38.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:39.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:39.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:00:39.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:00:39.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:39.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:39.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:39.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:39.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:39.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:39.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:39.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:39.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:39.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:39.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:39.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:39.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:39.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:40.349 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:40.349+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:40.366 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:40.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:40.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:40.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:40.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:40.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:40.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:40.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:40.414 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:40.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:41.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:41.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:41.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:41.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:41.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:41.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:41.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:41.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:41.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:41.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:42.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:42.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:42.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:42.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:42.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:42.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:42.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:42.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:42.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:42.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:43.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:43.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:43.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:43.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:43.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:43.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:43.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:43.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:43.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:43.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:43.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:43.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:43.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:43.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:43.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:43.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:43.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:43.621+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:43.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:43.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:43.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:43.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:43.645 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:43.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:43.645 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:43.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:44.239 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:00:44.239 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:00:44.240 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:00:44.249 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:00:45.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:45.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:45.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:45.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:45.656 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:45.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:45.659 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2305973382 2023-10-14T00:00:45.662 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:00:45.663 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241645 2023-10-14T00:00:45.664 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:00:45.664 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2305973382 ]] 2023-10-14T00:00:45.664 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2305973382 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\5\4\3\2\4\4\6\6\2 ]] 2023-10-14T00:00:45.664 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2305973382 2023-10-14T00:00:45.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:45.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:45.671 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:45.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:45.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:45.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:45.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:45.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:45.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:45.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:45.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:45.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:45.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:45.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:45.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:45.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:45.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:47.695 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2305973382 until 2023-10-14T01:00:46.901714+0000 (3600 sec) 2023-10-14T00:00:47.744 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:00:47.748 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2305973382 2023-10-14T00:00:47.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:47.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:47.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:47.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:47.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:47.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:48.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:48.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:48.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:48.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:48.093 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:48.093+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:48.102 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:48.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:48.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:48.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:48.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:48.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:48.139 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:48.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:48.973 INFO:tasks.workunit.client.0.smithi138.stderr:listed 4 entries 2023-10-14T00:00:49.026 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2305973382 2023-10-14T00:00:49.027 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:00:49.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:49.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:49.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:49.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:49.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:49.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:49.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:49.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:49.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:49.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:50.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:50.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:50.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:50.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:50.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:50.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:50.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:50.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:50.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:50.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:50.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:50.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:50.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:50.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:51.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:51.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:51.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:51.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:51.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:51.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:51.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:51.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:51.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:51.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:51.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:51.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:51.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:51.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:54.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:54.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:54.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:54.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:54.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:54.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:54.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:54.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:54.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:54.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:54.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:54.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:55.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:55.725+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:55.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:55.737+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:55.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:55.745+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:55.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:55.745+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:55.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:55.745+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:55.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:55.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:55.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:55.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:55.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:55.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:55.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:55.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:55.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:55.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:55.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:55.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:55.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:55.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:56.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:56.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:57.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:00:57.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:57.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:57.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:00:57.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:57.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:57.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:57.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:57.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:57.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:00:58.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:58.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:58.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:00:58.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:58.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.309+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:58.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:58.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:58.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:00:58.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:00:58.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:00:58.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:00:58.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:00:58.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:00:58.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:00:58.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:58.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:00:58.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:58.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:58.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:58.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:58.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:58.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:00:58.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:00:58.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:58.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:00:59.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:59.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:00:59.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:00:59.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:00:59.038 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:00:59.038 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:00:59.044 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:00:59.063 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:01:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.013+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.013+0000 7f082630d640 -1 librbd::ImageState: 0x5627d870fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.020 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.017+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.020 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.017+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cef00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.021 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.021+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.021 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.021+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8465c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.036 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.037+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.037 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.037+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86ce700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.061+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.061+0000 7f082630d640 -1 librbd::ImageState: 0x5627d87d6600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.069+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.069+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.069+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.069+0000 7f082630d640 -1 librbd::ImageState: 0x5627d85ca880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.069+0000 7f082630d640 -1 librbd::ImageState: 0x5627d870e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.069+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86ced00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.085+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.085+0000 7f082630d640 -1 librbd::ImageState: 0x5627d87d7700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.093+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.093+0000 7f082630d640 -1 librbd::ImageState: 0x5627d87d7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.101 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.101+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.101 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.101+0000 7f082630d640 -1 librbd::ImageState: 0x5627d87d7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.109+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.109+0000 7f082630d640 -1 librbd::ImageState: 0x5627d83c4900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.113+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.113+0000 7f082630d640 -1 librbd::ImageState: 0x5627d87d7900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.119 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.117+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.119 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.117+0000 7f082630d640 -1 librbd::ImageState: 0x5627d87d6880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.123 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.121+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.123 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.121+0000 7f082630d640 -1 librbd::ImageState: 0x5627d87d7d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.153+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.158 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.153+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7cc3e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.158 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.158 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.158 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.158 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.159 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8878080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8878280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8878480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:01:00.157+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8878680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:01:00.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:00.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:00.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:00.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:00.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:00.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:00.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:00.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:00.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:00.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:00.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:00.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:00.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:00.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:00.474 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2305973382 2023-10-14T00:01:00.480 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:01:00.481 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241660 2023-10-14T00:01:00.481 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:01:00.481 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2305973382 ]] 2023-10-14T00:01:00.481 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2305973382 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\0\5\9\7\3\3\8\2 ]] 2023-10-14T00:01:00.482 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:01:01.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:01.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:01.713 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:01.713+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:01.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:01.729+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:01.741 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:01.741+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:01.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:01.741+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:01.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:01.741+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:02.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:02.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:02.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:02.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:02.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:03.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:03.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:03.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:03.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:03.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:03.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:03.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:03.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:03.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:03.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:03.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:03.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:03.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:03.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:04.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:04.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:04.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:04.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:04.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:04.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:04.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:04.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:04.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:04.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:07.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:07.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:07.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:07.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:07.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:07.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:08.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:08.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:08.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:08.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:08.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:08.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:08.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:08.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:08.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:08.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:08.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:08.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:08.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:10.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:10.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:10.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:10.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:10.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:10.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:10.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:10.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:10.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:10.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:10.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:10.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:10.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:10.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:10.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:10.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:10.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:10.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:10.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:10.497 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:01:10.499 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:01:10.505 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:01:10.513 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:01:10.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:10.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:10.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:10.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:10.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:11.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:11.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:11.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:11.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:11.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:11.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:11.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:11.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:11.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:11.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:11.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:11.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:11.845 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2305973382 2023-10-14T00:01:11.856 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:01:11.864 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241671 2023-10-14T00:01:11.864 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:01:11.864 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2305973382 ]] 2023-10-14T00:01:11.865 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2305973382 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\0\5\9\7\3\3\8\2 ]] 2023-10-14T00:01:11.865 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:01:11.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:01:11.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:11.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:11.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:11.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:11.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:11.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:11.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:11.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:13.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:13.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:13.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:13.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:13.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:13.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:13.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:13.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:13.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:13.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:13.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:13.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:14.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:14.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:14.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:14.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:14.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:14.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:14.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:14.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:14.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:14.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:14.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:14.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:15.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:15.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:15.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:15.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:15.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:15.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:15.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:15.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:15.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:15.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:15.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:15.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:15.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:15.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:15.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:15.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:15.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:16.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:16.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:16.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:16.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:16.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:16.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:16.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:16.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:16.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:16.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:16.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:16.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:16.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:16.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:16.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:16.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:18.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:18.732+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:18.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:18.752+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:19.460 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.456+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:01:19.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:19.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:19.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:19.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:19.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:19.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:19.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:19.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:19.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:19.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.764+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:19.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.764+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:19.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:19.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:19.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:19.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:20.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:20.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:20.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:20.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:20.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:20.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:20.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:20.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:20.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:20.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:20.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:20.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:20.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:20.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:21.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:21.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:21.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:21.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:21.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:21.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:21.870 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:01:21.870 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:01:21.871 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:01:21.875 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:01:22.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:22.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:22.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:22.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:22.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:22.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:22.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:22.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:22.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:22.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:22.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:22.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:22.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:22.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:23.331 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2305973382 2023-10-14T00:01:23.332 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:01:23.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241683 2023-10-14T00:01:23.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:01:23.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2305973382 ]] 2023-10-14T00:01:23.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2305973382 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\0\5\9\7\3\3\8\2 ]] 2023-10-14T00:01:23.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:01:23.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:23.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:23.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:23.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:23.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:23.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:24.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:24.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:24.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:24.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:24.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:24.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:24.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:24.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:24.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:24.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:24.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:24.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:25.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:25.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:25.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:25.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:25.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:25.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:25.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:25.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:25.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:25.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:25.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:25.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:26.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:26.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:26.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:26.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:26.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:26.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:27.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:27.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:27.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:27.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:27.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:27.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:27.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:27.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:27.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:27.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:27.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:27.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:27.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:27.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:27.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:27.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:28.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:28.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:28.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:28.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:28.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:28.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:28.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:28.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:28.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:01:28.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:01:28.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:28.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:28.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:28.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:28.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:28.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:28.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:28.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:28.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:28.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:28.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:28.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:28.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:28.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:28.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:30.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:30.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:30.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:30.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:30.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:30.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:30.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:30.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:30.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:30.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:30.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:30.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:31.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:31.652+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:31.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:31.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:31.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:31.684+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:31.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:31.692+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:31.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:31.692+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:31.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:31.692+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:32.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:32.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:32.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:32.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:32.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:32.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:32.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:32.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:33.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:33.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:33.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:33.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:33.335 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:01:33.338 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:01:33.339 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:01:33.372 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:01:34.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:01:34.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:34.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:34.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:34.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:34.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:34.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:34.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:01:34.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:34.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:34.789 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2305973382 2023-10-14T00:01:34.790 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:01:34.790 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241694 2023-10-14T00:01:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:01:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2305973382 ]] 2023-10-14T00:01:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2305973382 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\0\5\9\7\3\3\8\2 ]] 2023-10-14T00:01:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:01:34.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:34.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:34.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:34.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:34.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:34.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:34.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:35.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:35.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:35.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:35.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:35.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:35.383 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:35.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:35.383 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:35.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:35.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:35.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:35.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:35.397 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:35.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:35.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:35.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:35.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:35.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:36.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:36.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:36.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:36.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:36.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:36.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:36.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:36.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:36.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:36.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:36.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:36.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:36.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:36.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:36.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:36.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:36.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:36.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:36.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:37.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:37.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:37.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:37.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:37.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:37.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:37.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:37.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:38.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:38.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:38.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:38.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:38.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:38.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:38.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:38.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:38.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:38.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:38.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:38.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:38.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:38.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:39.279 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:39.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:39.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:39.299 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:39.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:39.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:39.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:39.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:39.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:39.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:39.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:39.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:39.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.708+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:39.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.708+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:39.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.720+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:39.738 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:39.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:39.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:39.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:39.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:39.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:39.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:39.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:39.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:39.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:39.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:39.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:39.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:40.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:40.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:40.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:40.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:40.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:40.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:40.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:40.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:40.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:40.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:40.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:01:40.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:40.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:40.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:40.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:40.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:40.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:40.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:40.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:40.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:41.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:41.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:41.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:41.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:41.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:41.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:41.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:41.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:41.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:41.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:41.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:42.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:42.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:42.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:42.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:42.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:42.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:42.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:42.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:42.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:44.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:44.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:44.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:44.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:44.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:44.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:44.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:44.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:44.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:01:44.816 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:01:44.819 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:01:44.820 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:01:45.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:45.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:45.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:45.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:45.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:45.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:45.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:45.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:46.027 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2305973382 2023-10-14T00:01:46.028 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:01:46.029 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241706 2023-10-14T00:01:46.029 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:01:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2305973382 ]] 2023-10-14T00:01:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2305973382 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\0\5\9\7\3\3\8\2 ]] 2023-10-14T00:01:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:01:46.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:46.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:46.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:46.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:46.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:46.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:46.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:46.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:46.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:46.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:46.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:46.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:46.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:46.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:46.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:46.314 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.312+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:46.325 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:46.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:46.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:46.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:46.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:46.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:46.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:46.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:47.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:47.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:48.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:48.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:48.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:48.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:48.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:48.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:48.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:48.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:48.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:48.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:48.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:48.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:48.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:48.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:49.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:49.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:49.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:49.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:49.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:49.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:49.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.740+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:49.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.740+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:49.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:49.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:49.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:49.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:49.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:49.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:49.789 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.788+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:49.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:49.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:49.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:49.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:49.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.796+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:49.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:49.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:49.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:49.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:50.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:50.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:50.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:50.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:50.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:50.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:50.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:50.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:50.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:50.820 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:50.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:50.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:50.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:50.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:50.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:50.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:50.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:50.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:50.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:50.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:50.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:50.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:50.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:50.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:50.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:50.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:51.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:51.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:51.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:51.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:51.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:51.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:51.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:51.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:51.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:51.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:51.744+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:52.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:52.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:52.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:52.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:52.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:52.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:52.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:52.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:52.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:52.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:52.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:52.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:52.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:52.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:52.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:52.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:52.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:52.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:52.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:52.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:52.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:52.776 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:52.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:52.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:52.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:52.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:52.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:52.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:52.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:52.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:52.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:53.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:01:53.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:53.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:53.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.668+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:53.695 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.692+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:53.696 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.692+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:53.696 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.692+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:53.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:53.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:53.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:53.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:53.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:53.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:53.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:53.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:53.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:53.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:53.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:53.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:53.802 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.800+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:53.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:53.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:54.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:54.088+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:55.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:55.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:01:55.158 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:55.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:01:55.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:55.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:01:55.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:55.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:01:55.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:55.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:01:55.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:01:55.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:01:56.035 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:01:56.035 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:01:56.036 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:01:56.037 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:01:57.278 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2845967498 2023-10-14T00:01:57.278 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:01:57.279 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241717 2023-10-14T00:01:57.279 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:01:57.279 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2845967498 ]] 2023-10-14T00:01:57.280 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2845967498 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\0\5\9\7\3\3\8\2 ]] 2023-10-14T00:01:57.280 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2845967498 2023-10-14T00:01:57.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:01:57.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:01:57.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:01:57.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:57.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:01:57.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:57.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:57.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:57.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:57.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:57.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:58.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:58.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:01:58.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:58.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:01:58.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:58.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:58.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:58.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:59.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:59.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:01:59.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:59.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:01:59.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:59.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:01:59.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:59.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:01:59.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:01:59.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:01:59.753 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2845967498 until 2023-10-14T01:01:59.091314+0000 (3600 sec) 2023-10-14T00:01:59.814 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2845967498 2023-10-14T00:01:59.831 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:02:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.004+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.008 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.004+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cfa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.012+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.012+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7cc3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.012+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.012+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.024+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.024+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7cc2180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896ca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.047 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.044+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.044+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.044+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.044+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.048+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.049 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.048+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.092 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.088+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.092 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.088+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.149 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.148+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.149 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.148+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.153 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.152+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.153 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.152+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.154 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.152+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.154 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.152+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.169 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.168+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.170 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.168+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.193 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.192+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.193 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.192+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896dc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.214 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.214 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.215 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.215 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.215 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.215 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aae080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.215 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aae280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.216 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.212+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aae480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.235 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.232+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.236 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.232+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aaee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.240 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.240+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.241 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:02:00.240+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aaed00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:02:00.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:00.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:00.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:00.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:00.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:00.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:01.090 INFO:tasks.workunit.client.0.smithi138.stderr:listed 5 entries 2023-10-14T00:02:01.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:01.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:01.143 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2845967498 2023-10-14T00:02:01.143 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:02:01.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:01.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:01.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:01.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:01.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:01.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:01.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:01.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:01.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:01.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:01.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:01.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:01.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:01.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:01.922 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:02:01.920+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:02:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:02.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:02.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:02.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:02.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:02.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:02.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:02.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:03.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:03.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:03.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:03.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:03.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:03.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:03.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:03.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:03.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:03.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:03.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:03.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:03.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:03.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:04.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:04.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:04.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:04.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:04.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:04.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:04.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:04.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:05.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.228+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:05.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.228+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:05.241 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:05.242 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:05.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:05.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:05.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:05.571 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:05.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.572+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:05.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:05.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:05.588+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:06.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:06.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:06.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:06.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:06.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:06.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:06.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:06.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:06.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:06.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:06.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:06.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:06.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:06.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:07.240 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:07.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:07.241 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:07.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:07.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:07.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:07.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:07.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:07.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:07.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:07.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:07.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:07.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:07.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:07.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:07.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:07.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:07.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:07.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:07.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:07.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:07.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:07.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:07.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:09.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:09.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:09.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:09.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:09.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:09.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:09.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:09.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:10.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:10.512+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:10.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:10.532+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:10.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:10.544+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:10.561 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:10.556+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:10.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:10.556+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:10.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:10.556+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:11.144 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:02:11.145 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:02:11.148 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:02:11.162 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:02:12.579 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2845967498 2023-10-14T00:02:12.591 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:02:12.593 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241732 2023-10-14T00:02:12.593 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:02:12.593 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2845967498 ]] 2023-10-14T00:02:12.593 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2845967498 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\4\5\9\6\7\4\9\8 ]] 2023-10-14T00:02:12.594 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:02:13.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:13.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:13.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:13.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:13.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:13.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:13.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:13.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:13.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:13.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:13.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:13.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:14.286 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:14.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:14.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:14.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:14.313 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:14.312+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:14.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:14.328+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:14.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:14.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:14.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:14.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:14.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:14.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:16.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:16.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:16.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:16.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:18.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:18.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:18.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:18.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:18.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:18.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:20.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:20.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:20.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:20.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:20.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:20.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:20.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:20.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:20.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:20.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:20.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:20.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:21.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:21.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:21.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:21.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:21.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:21.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:21.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:21.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:22.607 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:02:22.618 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:02:22.622 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:02:22.623 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:02:23.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:23.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:23.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:23.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:23.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:23.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:23.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:23.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:23.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:23.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:23.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:23.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:23.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:23.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:23.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:23.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:23.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:23.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:23.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:23.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:23.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:23.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:23.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:24.129 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2845967498 2023-10-14T00:02:24.129 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:02:24.130 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241744 2023-10-14T00:02:24.130 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:02:24.130 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2845967498 ]] 2023-10-14T00:02:24.130 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2845967498 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\4\5\9\6\7\4\9\8 ]] 2023-10-14T00:02:24.131 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:02:24.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:24.584+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:24.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:24.584+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:24.587 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:24.584+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:24.593 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:24.592+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:24.600 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:24.600+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:24.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:24.600+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:24.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:24.600+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:25.088 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:25.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:25.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:25.088+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:27.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:27.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:27.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:27.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:27.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:27.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:27.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:27.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:27.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:27.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:27.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:27.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:27.212 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:27.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:27.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:27.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:27.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:27.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:27.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:27.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:27.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:27.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:29.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:29.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:30.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:30.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:30.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:30.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:30.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:30.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:30.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:30.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:30.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:30.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:30.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:32.738 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:32.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:32.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:32.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:32.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:32.975 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:32.975 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:32.975 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:32.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:34.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:34.142 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:02:34.142 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:02:34.143 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:02:34.148 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:02:34.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:34.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:34.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:34.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:34.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:34.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:34.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:34.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:34.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:34.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:34.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:34.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:34.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:34.524+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:34.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:34.536+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:34.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:34.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:34.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:34.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:34.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:34.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:34.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:34.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:34.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:34.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:34.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:34.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:34.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:34.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:35.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:35.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:35.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:35.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:35.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:35.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:35.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:35.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:35.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:35.481 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2845967498 2023-10-14T00:02:35.481 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:02:35.482 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241755 2023-10-14T00:02:35.482 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:02:35.482 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2845967498 ]] 2023-10-14T00:02:35.483 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2845967498 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\4\5\9\6\7\4\9\8 ]] 2023-10-14T00:02:35.483 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:02:38.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:38.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:38.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:38.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:38.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:38.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:38.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:38.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.192+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:38.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.192+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:38.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:38.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:38.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:38.239 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:38.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:38.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:38.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:38.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:38.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.668+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:38.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:38.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:38.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:38.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:38.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:38.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:38.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:38.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:39.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:39.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:39.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:39.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:39.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:39.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:39.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:39.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:39.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:42.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:42.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:42.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:42.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:42.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:42.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:42.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:42.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:42.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:42.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:42.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:42.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:42.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:42.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:42.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:42.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:43.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:43.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:43.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:43.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:43.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:43.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:43.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:43.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:43.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:43.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:43.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:43.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:44.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:44.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:44.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:44.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:44.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:44.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:44.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:44.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:44.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:44.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:44.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:44.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:45.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:45.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:45.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.175+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:45.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.191+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:45.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:45.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:45.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:45.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:45.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.371+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:45.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.371+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:45.373 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:45.371+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:45.485 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:02:45.485 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:02:45.486 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:02:45.502 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:02:47.011 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2845967498 2023-10-14T00:02:47.011 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:02:47.012 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241767 2023-10-14T00:02:47.012 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:02:47.012 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2845967498 ]] 2023-10-14T00:02:47.013 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2845967498 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\4\5\9\6\7\4\9\8 ]] 2023-10-14T00:02:47.013 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:02:47.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:47.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:47.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:47.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:47.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:47.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:47.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:47.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:47.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:47.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:47.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:47.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:48.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:48.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:48.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:48.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:48.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:48.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:48.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:48.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:48.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:02:48.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:48.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:48.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:48.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:48.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:48.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:48.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:48.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:48.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:49.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:49.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:49.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:49.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:49.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:49.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:49.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:49.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:49.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:49.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:49.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:49.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:51.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:51.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:51.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:51.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:51.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:51.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:51.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:51.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:51.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:51.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:51.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:51.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:51.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:51.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:51.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:51.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:51.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:51.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:51.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:51.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:51.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:51.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:51.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:51.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:51.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:51.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:51.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:52.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:52.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:52.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:52.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:52.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:52.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:52.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:52.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:52.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:52.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:52.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:52.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:53.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:53.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:53.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:53.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:53.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:53.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:53.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:53.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:53.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:53.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:53.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:53.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:53.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:53.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:54.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:02:54.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:54.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:54.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:54.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:54.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:54.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:54.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:54.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:54.599+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:54.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:54.599+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:54.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:54.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:54.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:54.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:54.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:54.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:54.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:54.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:54.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:54.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:56.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:56.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:56.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:56.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:56.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:56.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:57.031 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:02:57.032 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:02:57.037 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:02:57.041 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:02:57.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:02:57.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:57.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:57.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:57.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:57.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:57.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:57.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:57.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:57.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:57.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:57.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:57.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:57.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:57.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:57.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:57.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:57.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:02:57.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:57.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:57.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:57.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:58.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:57.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:58.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:58.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:58.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:58.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:02:58.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:58.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:02:58.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:58.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:02:58.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:58.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:02:58.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:58.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:02:58.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:58.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:02:58.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:02:58.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:02:58.466 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2845967498 2023-10-14T00:02:58.466 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:02:58.467 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241778 2023-10-14T00:02:58.467 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:02:58.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2845967498 ]] 2023-10-14T00:02:58.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2845967498 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\4\5\9\6\7\4\9\8 ]] 2023-10-14T00:02:58.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:02:59.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:02:59.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:59.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:59.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:59.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:59.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:59.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:02:59.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:02:59.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:02:59.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:02:59.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:02:59.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:02:59.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:02:59.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:00.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:00.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:00.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:00.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:00.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:00.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:00.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:00.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:00.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:00.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:00.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:00.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:00.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:00.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:01.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:01.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:01.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:01.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:01.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:01.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:01.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:02.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:02.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:02.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:02.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:02.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:02.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:02.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:02.325 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:02.323+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:02.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:02.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:02.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:02.343+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:02.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:02.343+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:02.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:02.343+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:02.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:03:02.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:02.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:02.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:02.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:02.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:02.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:02.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:02.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:03.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:03.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.915+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:03.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.915+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:03.931 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.927+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:03.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.931+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:03.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:03.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:03.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:03.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:04.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:04.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:04.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:04.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:04.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:04.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:04.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:04.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:04.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:04.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:04.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:04.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:04.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:04.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:05.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:05.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:05.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:05.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:05.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:05.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:05.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:05.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:05.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:05.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:05.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:06.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:06.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:06.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:06.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:06.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:06.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:06.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:06.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:06.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:06.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:06.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:06.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:06.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:06.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:06.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:06.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:08.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:03:08.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:08.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:08.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:08.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:08.472 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:03:08.472 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:03:08.473 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:03:08.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:08.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:08.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:08.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:08.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:08.485 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:03:09.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:09.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:09.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:09.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:09.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:09.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:09.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:09.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:09.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:09.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:09.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:09.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:10.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2993341611 2023-10-14T00:03:10.251 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:03:10.252 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241790 2023-10-14T00:03:10.252 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:03:10.253 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2993341611 ]] 2023-10-14T00:03:10.253 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2993341611 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\4\5\9\6\7\4\9\8 ]] 2023-10-14T00:03:10.253 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2993341611 2023-10-14T00:03:10.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:10.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:10.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:10.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:10.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:10.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:10.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:10.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:11.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:11.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:11.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:11.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:11.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:11.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:12.749 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2993341611 until 2023-10-14T01:03:12.084149+0000 (3600 sec) 2023-10-14T00:03:12.796 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2993341611 2023-10-14T00:03:12.805 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:03:13.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:13.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:13.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:13.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:13.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:13.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:13.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:13.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:13.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:13.991 INFO:tasks.workunit.client.0.smithi138.stderr:listed 6 entries 2023-10-14T00:03:14.066 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2993341611 2023-10-14T00:03:14.066 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:03:14.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:03:14.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:14.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:14.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:14.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:14.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:14.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:14.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:14.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:14.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:14.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.487+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:14.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.487+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:14.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:14.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:14.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:14.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:14.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:14.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.843+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:14.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:14.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:14.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:14.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:15.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:15.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:15.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:15.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:15.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:15.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:15.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:15.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:03:15.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:03:15.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:15.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:15.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:15.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:16.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:16.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:16.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:16.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:16.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:16.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:16.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:16.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:16.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:16.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:16.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:16.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:16.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:16.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:17.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:17.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:17.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:17.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:17.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:17.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:17.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:17.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:17.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:17.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:17.643 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:17.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:17.672 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:17.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:18.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:18.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:18.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:18.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:18.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:18.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:18.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:18.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:18.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:18.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:18.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:18.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:18.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:18.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:18.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:18.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:18.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:18.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:20.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:20.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:20.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:20.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:20.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:03:20.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:03:20.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:20.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:20.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:20.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:20.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:20.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:20.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:20.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:20.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:20.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:21.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:21.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:21.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:21.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:22.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:22.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:22.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:22.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:22.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:22.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:22.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:22.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:22.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:22.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:22.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:22.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:22.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:22.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:22.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:24.089 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:03:24.089 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:03:24.089 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:03:24.090 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:03:24.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:24.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:24.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:24.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:24.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:24.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:24.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:24.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:24.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:24.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:03:24.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:24.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:24.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:24.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:24.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:24.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:24.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:24.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:25.552 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2993341611 2023-10-14T00:03:25.552 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:03:25.553 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241805 2023-10-14T00:03:25.553 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:03:25.553 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2993341611 ]] 2023-10-14T00:03:25.554 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2993341611 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\9\3\3\4\1\6\1\1 ]] 2023-10-14T00:03:25.554 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:03:26.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:26.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:26.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:26.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:26.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:26.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:26.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:26.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:26.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:26.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:26.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:26.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:29.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:29.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:29.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:29.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:29.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:29.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:29.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:29.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:29.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:29.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:29.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:29.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:31.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:31.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:31.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:31.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:31.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:31.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:31.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:31.539+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:31.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:31.555+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:31.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:31.555+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:31.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:31.555+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:32.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:32.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:32.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:32.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:32.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:32.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:32.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:32.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:32.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:32.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:32.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:32.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:32.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:32.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:33.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:33.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:33.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:33.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:33.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:33.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:33.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:33.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:33.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:33.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:34.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:34.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:35.553 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:03:35.553 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:03:35.561 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:03:35.572 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:03:36.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:36.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:03:36.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:36.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:36.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:36.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:36.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:36.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:36.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:36.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:36.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:36.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:36.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:36.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:36.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:36.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:36.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:36.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:36.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:36.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:37.111 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2993341611 2023-10-14T00:03:37.118 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:03:37.123 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241817 2023-10-14T00:03:37.124 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:03:37.124 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2993341611 ]] 2023-10-14T00:03:37.124 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2993341611 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\9\3\3\4\1\6\1\1 ]] 2023-10-14T00:03:37.124 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:03:37.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:37.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:37.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:37.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:37.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:37.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:39.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:39.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:39.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:39.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:39.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:39.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:39.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:39.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:39.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:39.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:39.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:39.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:39.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:40.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:40.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:40.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:40.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:40.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:40.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:41.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:41.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:41.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:41.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:41.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:41.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:41.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:41.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:41.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:41.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:41.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:41.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:41.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:41.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:42.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:42.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:42.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:42.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:42.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:42.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:42.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:42.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:44.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:44.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:44.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:44.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:44.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:44.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:44.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:44.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:44.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:47.125 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:03:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:03:47.129 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:03:47.130 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:03:48.431 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2993341611 2023-10-14T00:03:48.431 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:03:48.432 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241828 2023-10-14T00:03:48.432 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:03:48.432 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2993341611 ]] 2023-10-14T00:03:48.432 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2993341611 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\9\3\3\4\1\6\1\1 ]] 2023-10-14T00:03:48.433 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:03:48.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:48.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:48.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:48.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:48.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:48.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:49.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:49.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:49.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:49.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:49.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:49.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:49.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:49.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:49.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:49.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:49.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:49.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:49.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:49.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:50.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:50.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:50.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:50.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:50.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:50.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:50.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:50.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:51.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:51.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:51.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:51.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:52.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:52.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:03:52.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:52.451+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:03:52.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:52.451+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:03:52.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:52.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:03:52.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:52.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:03:52.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:03:52.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:03:53.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:53.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:53.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:53.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:53.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:53.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:53.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:53.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:53.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:53.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:53.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:53.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:53.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:53.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:56.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:56.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:56.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:56.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:56.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:56.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:56.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:56.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:56.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:56.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:56.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:56.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:56.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:56.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:56.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:56.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:56.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:56.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:56.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:56.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:56.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:56.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:56.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:56.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:56.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:56.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:58.436 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:03:58.436 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:03:58.436 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:03:58.454 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:03:59.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:59.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:59.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:59.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:59.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:59.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:03:59.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:59.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:03:59.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:59.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:59.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:59.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:59.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:59.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:03:59.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:03:59.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:03:59.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:03:59.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:03:59.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:03:59.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:03:59.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.011+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.011+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a8c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.023+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.023+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.023+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.023+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a8e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.025 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.023+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.025 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.023+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.027+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.027+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.043+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.043+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.050 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.047+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.050 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.047+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.075+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.075+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.091+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.091+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.104 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2993341611 2023-10-14T00:04:00.104 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:04:00.106 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241840 2023-10-14T00:04:00.106 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:04:00.106 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2993341611 ]] 2023-10-14T00:04:00.106 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2993341611 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\9\3\3\4\1\6\1\1 ]] 2023-10-14T00:04:00.106 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:04:00.124 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.123+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.124 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.123+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.126 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.123+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.127 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.123+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.127+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.127+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.142 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.139+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.142 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.139+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.155+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.155+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.200 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.199+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.202 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.199+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.235 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.235+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.235 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.235+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.263 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.259+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.263 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.259+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.263 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.259+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.264 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.259+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.264 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.259+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.265 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.259+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b61000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.282 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.279+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:00.282 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:04:00.279+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b61200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:04:01.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:01.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:01.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:01.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:01.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:01.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:01.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:01.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:01.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:01.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:01.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:01.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:04:01.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:04:01.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:01.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:01.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:01.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:01.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:01.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:03.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:04:03.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:04:03.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:03.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:03.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:03.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:03.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:03.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:03.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:03.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:03.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:04.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:04.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:04.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:04.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:04.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:04.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:04.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:04.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:04.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:04.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:04.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:04.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:04.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:04.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:06.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:06.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:06.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:06.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:06.667 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:06.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:06.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:06.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:06.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:06.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:06.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:06.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:06.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:06.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:07.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:07.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:07.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:07.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:07.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:07.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:07.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:07.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:07.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:07.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:08.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:08.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:08.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:08.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:08.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:08.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:08.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:08.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:08.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:08.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:08.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:08.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:08.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:08.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:08.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:08.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:09.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:09.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:09.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:09.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:09.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:09.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:09.008 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:09.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:09.008 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:09.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:09.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:09.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:09.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:09.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:09.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:09.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:09.732 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:09.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:09.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:09.742+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:10.109 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:04:10.113 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:04:10.116 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:04:10.134 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:04:10.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:10.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:10.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:10.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.374+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:10.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.374+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:10.379 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.374+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:10.379 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.374+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:10.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:10.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:10.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:10.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:10.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:10.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:10.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:10.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:10.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:10.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:10.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:10.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:10.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:10.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:10.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:10.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:10.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:10.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:10.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:10.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:10.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:11.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:11.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:11.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:11.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:11.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:11.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:11.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:11.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:11.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:11.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:11.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:11.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:11.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:11.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:11.705 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2993341611 2023-10-14T00:04:11.705 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:04:11.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241851 2023-10-14T00:04:11.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:04:11.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2993341611 ]] 2023-10-14T00:04:11.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2993341611 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\9\3\3\4\1\6\1\1 ]] 2023-10-14T00:04:11.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:04:13.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:13.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:04:14.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:13.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:14.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:13.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:14.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:14.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:14.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:14.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:14.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:14.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:14.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:04:14.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:04:14.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:14.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:14.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:14.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:14.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:14.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:14.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:14.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:14.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:14.314 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:14.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:14.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:14.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:14.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:14.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:14.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:14.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:14.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:14.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:14.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:14.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:15.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:15.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:15.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:15.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:15.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:15.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:16.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:16.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:16.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:16.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:16.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:16.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:16.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:16.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:16.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:17.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:17.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:17.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:17.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:17.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:17.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:17.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:17.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:17.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:17.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:17.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:17.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:17.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:17.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:17.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:17.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:17.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:17.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:17.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:17.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:17.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:17.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:17.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:17.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:18.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:18.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:18.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:18.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:18.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:18.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:18.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:18.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:18.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:18.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:19.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:19.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:19.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:19.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:19.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:19.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:19.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:19.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:19.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:19.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:19.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:19.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:19.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:20.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:20.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:20.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:20.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:20.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:20.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:20.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:20.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:20.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:20.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:20.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:20.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:21.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:21.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:21.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:21.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:21.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:21.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:21.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:21.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:21.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:21.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:21.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:21.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:21.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:21.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:21.723 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:04:21.724 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:04:21.724 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:04:21.737 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:04:22.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:22.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:22.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:22.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:22.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:22.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:22.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:22.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:22.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:22.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:22.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:22.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:22.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:22.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:22.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:22.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:22.466 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:22.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:22.466 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:22.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:22.467 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:22.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:23.138 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/92831769 2023-10-14T00:04:23.138 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:04:23.148 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241863 2023-10-14T00:04:23.148 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:04:23.149 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/92831769 ]] 2023-10-14T00:04:23.149 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/92831769 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\9\3\3\4\1\6\1\1 ]] 2023-10-14T00:04:23.149 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/92831769 2023-10-14T00:04:23.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:23.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:23.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:23.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:23.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:23.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:23.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:23.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:23.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:24.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:04:24.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:24.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.618+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:24.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:24.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:24.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:24.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:24.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:24.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:24.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:25.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:25.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:25.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:25.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:25.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:25.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:25.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:25.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:25.749 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/92831769 until 2023-10-14T01:04:25.130731+0000 (3600 sec) 2023-10-14T00:04:25.839 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:04:25.840 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/92831769 2023-10-14T00:04:26.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:26.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:26.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:26.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:26.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:26.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:26.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:26.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:26.509 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:26.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:26.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:26.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:26.580 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:26.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:26.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:26.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:26.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:26.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:26.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:26.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:26.959 INFO:tasks.workunit.client.0.smithi138.stderr:listed 7 entries 2023-10-14T00:04:27.018 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/92831769 2023-10-14T00:04:27.019 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:04:27.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:27.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:27.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:27.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:27.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:27.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:27.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:27.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:27.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:27.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:27.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:27.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:28.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:28.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:28.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:28.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:28.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:28.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:28.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:28.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:28.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:28.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:28.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:28.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:30.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:30.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:30.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:30.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:30.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:30.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:30.597 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:30.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:30.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:30.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:30.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:30.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:30.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:30.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:31.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:31.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:31.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:31.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:31.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:31.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:31.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:31.267 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:31.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:31.267 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:31.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:31.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:31.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:31.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:31.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:31.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:31.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:31.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:32.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:32.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:32.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:32.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:32.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:32.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:32.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:32.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:32.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:32.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:32.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:32.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:32.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:32.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:32.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:33.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:33.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:33.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:33.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:33.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:33.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:33.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:33.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:04:33.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:04:33.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:33.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:33.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:33.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:33.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:33.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:33.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:37.041 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:04:37.046 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:04:37.053 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:04:37.086 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:04:37.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:37.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:37.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:37.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:37.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:37.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:37.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:37.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:37.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:37.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:37.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:37.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:38.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:38.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:38.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:38.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:38.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:38.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:38.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:38.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:38.528 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/92831769 2023-10-14T00:04:38.528 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:04:38.529 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241878 2023-10-14T00:04:38.529 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:04:38.529 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/92831769 ]] 2023-10-14T00:04:38.529 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/92831769 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\2\8\3\1\7\6\9 ]] 2023-10-14T00:04:38.530 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:04:39.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:39.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:40.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:40.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:40.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:40.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:40.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:40.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:40.240 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:40.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:40.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:40.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:40.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:40.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:40.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:40.626 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.622+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:40.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:40.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:40.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:40.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:41.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:41.109 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.106+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:41.110 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.106+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:41.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:41.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:41.124 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:41.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:41.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:41.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:41.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:41.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:41.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:41.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:41.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:41.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:41.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:41.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:41.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:41.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:41.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:41.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:41.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:41.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:41.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:41.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:41.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:04:41.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:41.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:41.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:41.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:41.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:41.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:41.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:41.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:42.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:42.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:42.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:42.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:42.377 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:42.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:42.377 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:42.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:42.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:42.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:42.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:42.374+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:42.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:42.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:42.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:42.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:42.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:42.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:42.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:42.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:42.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:42.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:42.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:43.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:43.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:43.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:43.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:43.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:43.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:43.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:43.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:43.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:43.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:43.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:43.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:44.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:44.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:44.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:44.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:44.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:44.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:44.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:44.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:44.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:44.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:44.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:44.294+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:44.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:44.294+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:44.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:44.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:44.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:44.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:44.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:44.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:44.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:44.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:44.338 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:44.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:46.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:46.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:46.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:46.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:46.869 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:46.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:46.870 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:46.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:46.870 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:46.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:48.530 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:04:48.531 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:04:48.541 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:04:48.541 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:04:48.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:48.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:49.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:04:49.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:49.940 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/92831769 2023-10-14T00:04:49.941 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:04:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241889 2023-10-14T00:04:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:04:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/92831769 ]] 2023-10-14T00:04:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/92831769 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\2\8\3\1\7\6\9 ]] 2023-10-14T00:04:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:04:49.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:49.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:49.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:49.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:49.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:49.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:49.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:50.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:50.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:50.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:50.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:50.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:50.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:50.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:50.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:50.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:50.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:50.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:50.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:50.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:50.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:51.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:51.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:04:51.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:51.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:51.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:51.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:51.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:51.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:51.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:51.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:51.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:51.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:51.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:51.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:52.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:52.854+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:52.857 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:52.854+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:52.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:52.870+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:52.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:52.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:52.884 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:52.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:52.884 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:52.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:52.885 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:52.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:53.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:53.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:53.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:53.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:53.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:53.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:53.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:53.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:54.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:54.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:04:54.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:54.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:54.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:54.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:56.946 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:56.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:56.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:56.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:56.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:56.970+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:56.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:56.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:56.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:56.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:56.984 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:56.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:57.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:57.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:57.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:57.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:57.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:57.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:57.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:57.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:57.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:57.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:58.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:58.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:04:58.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:58.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:04:58.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:58.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:04:58.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:58.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:04:58.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:04:58.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:04:59.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.234+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:59.241 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:59.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:59.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:59.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:59.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:04:59.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:04:59.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:04:59.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:04:59.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:04:59.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:04:59.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:04:59.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:04:59.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:04:59.949 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:04:59.949 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:04:59.949 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:04:59.962 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:05:00.016 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.014+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.017 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.014+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8bce100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.026 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.022+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.027 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.022+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bce300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.033 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.030+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.033 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.030+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bce500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.042+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.042+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bce700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.050 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.046+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.046+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bce900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.064 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.062+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.064 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.062+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bceb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.070+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.070+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcf100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.079 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.078+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.078+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcf680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcf500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcf300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bced80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.098+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcf900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.106+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.106+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcfb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.106+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.106+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcfd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.122 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.118+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.123 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.118+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bcff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.130+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.130+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfc100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.134+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.134+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfc300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.141 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.138+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.141 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.138+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfc500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.147 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.146+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:00.147 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:05:00.146+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfc700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:05:01.435 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/92831769 2023-10-14T00:05:01.436 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:05:01.438 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241901 2023-10-14T00:05:01.438 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:05:01.438 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/92831769 ]] 2023-10-14T00:05:01.438 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/92831769 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\2\8\3\1\7\6\9 ]] 2023-10-14T00:05:01.438 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:05:04.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:04.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:04.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:04.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:04.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:04.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:04.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:04.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:04.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:04.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:05.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:05.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:05.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:05.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:05.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:05.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:05.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:05:05.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:05.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:05.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:05.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:05.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:05.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:05.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:05.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:07.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:07.518+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:07.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:07.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:07.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:07.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:07.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:07.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:07.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:07.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:08.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:08.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:08.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:08.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:08.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:08.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:08.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:08.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:08.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:08.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:08.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:08.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:09.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:09.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:09.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:09.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:09.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:09.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:09.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:09.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:09.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:09.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:09.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:09.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:09.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:09.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:10.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:10.290+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:10.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:10.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:10.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:10.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:10.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:10.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:10.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:10.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:10.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:10.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:10.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:10.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:10.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:10.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:10.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:10.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:11.449 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:05:11.449 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:05:11.449 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:05:11.462 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:05:12.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:12.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:12.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:12.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:12.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:12.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:12.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:12.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:12.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:12.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:12.978 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/92831769 2023-10-14T00:05:12.981 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:05:12.982 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241912 2023-10-14T00:05:12.982 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:05:12.982 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/92831769 ]] 2023-10-14T00:05:12.982 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/92831769 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\2\8\3\1\7\6\9 ]] 2023-10-14T00:05:12.983 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:05:13.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:13.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:13.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:13.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:13.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:13.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:13.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:13.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:13.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:13.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:13.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:13.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:13.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:13.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:13.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:13.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:13.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:13.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:13.910+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:13.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:13.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:13.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:13.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:13.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:13.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:14.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:14.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:14.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:14.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:14.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:14.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:14.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:14.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:14.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:15.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.014+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:05:15.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:15.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:15.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:15.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:15.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:15.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:15.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:15.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.290+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:15.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.290+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:15.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.294+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:15.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:15.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:15.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:15.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:15.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:15.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:15.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:15.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:15.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:15.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:15.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:15.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:16.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:16.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:16.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:16.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:16.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:16.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:16.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:16.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:16.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:16.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:16.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:18.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:18.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:18.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:18.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:18.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:18.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:18.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:18.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:18.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:18.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:18.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:18.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:20.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:05:20.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:05:20.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:05:20.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:20.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:20.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:20.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:20.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:20.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:20.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:20.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:20.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:20.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:20.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:20.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:20.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:20.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:20.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:20.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:20.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:20.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:20.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:21.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:21.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:21.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:21.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:21.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:21.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:21.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:21.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:21.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:21.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:21.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:21.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:21.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:21.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:21.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:21.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:22.993 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:05:23.009 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:05:23.013 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:05:23.019 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:05:23.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:05:23.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:05:23.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:23.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:23.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:23.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:23.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:23.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:23.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:24.629 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/92831769 2023-10-14T00:05:24.629 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:05:24.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241924 2023-10-14T00:05:24.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:05:24.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/92831769 ]] 2023-10-14T00:05:24.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/92831769 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\2\8\3\1\7\6\9 ]] 2023-10-14T00:05:24.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:05:25.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:25.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.609+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:25.629 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:25.629 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:25.630 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:25.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:25.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:05:25.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:25.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:25.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:25.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:25.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:25.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:25.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:25.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:25.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:25.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:25.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:25.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:25.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:25.992 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:26.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.997+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:26.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.997+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:26.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:25.997+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:26.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:26.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:26.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:26.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:26.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:26.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:26.124 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:26.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:26.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:26.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:26.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:26.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:26.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:26.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:26.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:26.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:26.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:26.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:26.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:26.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:27.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:27.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:27.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:27.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:27.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:27.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:27.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:27.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:27.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:27.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:27.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:27.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:28.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:28.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:28.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:28.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:28.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:28.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:28.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:28.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:29.272 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:29.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:29.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:29.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:29.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:29.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:29.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:29.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:29.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:29.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:29.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:29.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:29.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:29.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:29.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:29.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:30.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:30.907 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:30.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.937+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:30.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.945+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:30.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:30.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:30.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:30.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:30.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:31.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:31.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:31.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:31.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:31.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:31.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:31.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:31.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:31.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:31.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:31.695 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:31.693+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:31.719 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:31.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:31.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:31.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:31.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:31.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:31.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:31.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:34.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:34.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:34.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:34.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:34.636 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:05:34.636 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:05:34.638 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:05:34.641 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:05:34.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:05:34.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:34.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:34.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:34.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:34.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:34.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:35.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:35.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:35.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:35.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:35.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:35.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:35.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:35.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:35.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:36.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:36.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:36.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:36.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1006714013 2023-10-14T00:05:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:05:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241936 2023-10-14T00:05:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:05:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1006714013 ]] 2023-10-14T00:05:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1006714013 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\2\8\3\1\7\6\9 ]] 2023-10-14T00:05:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/1006714013 2023-10-14T00:05:36.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:36.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:36.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:36.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:05:36.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:36.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:36.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:36.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:36.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:36.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:36.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:36.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:36.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:36.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:36.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:37.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:37.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.013+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:37.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:37.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:37.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:37.449 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:37.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:37.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:37.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:37.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:37.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:37.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:38.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:38.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:38.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:38.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:38.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:38.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:38.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:38.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:38.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:38.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:38.858 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/1006714013 until 2023-10-14T01:05:38.185482+0000 (3600 sec) 2023-10-14T00:05:38.902 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:05:38.908 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/1006714013 2023-10-14T00:05:40.048 INFO:tasks.workunit.client.0.smithi138.stderr:listed 8 entries 2023-10-14T00:05:40.152 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/1006714013 2023-10-14T00:05:40.152 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:05:40.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:40.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:40.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:40.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:40.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:40.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:40.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:40.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:40.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:40.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:40.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:40.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:42.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:42.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:42.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:42.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:42.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:42.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:42.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:43.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:42.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:43.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:43.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:43.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:43.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:43.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:43.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:43.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:43.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:43.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:43.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:43.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:43.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:43.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:43.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:43.671 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:43.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:43.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:43.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:43.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:43.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:43.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:43.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:43.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:43.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:44.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:44.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:44.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:44.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:44.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:44.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:44.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:44.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:44.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:44.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:44.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:44.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:44.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:45.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:45.761+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:45.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:45.761+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:45.795 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:45.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:45.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:45.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:45.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:45.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:45.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:45.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:47.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:47.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:47.185+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:47.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:47.189+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:47.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:47.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:47.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:47.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:47.205+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:47.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:47.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:47.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:47.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:47.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:47.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:47.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:47.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:47.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:47.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:47.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:48.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:48.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:48.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:48.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:48.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:48.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:48.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:48.517+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:48.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:48.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:48.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:48.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:48.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:48.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:05:48.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:48.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:48.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:48.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:48.573 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:48.569+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:48.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:48.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:48.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:48.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:48.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:48.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:49.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:49.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:49.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:49.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:49.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:49.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:49.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:49.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:49.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:49.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:49.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:49.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:49.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:49.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:05:49.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:49.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:49.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:49.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:49.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:49.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:49.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:49.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:49.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:49.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:49.692 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:49.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:50.155 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:05:50.164 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:05:50.165 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:05:50.166 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:05:50.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:50.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:50.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:50.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:50.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:50.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:50.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:50.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:50.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:50.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:50.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:50.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:50.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:50.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:50.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:50.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:50.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:50.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:50.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:50.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:50.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:50.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:51.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:51.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:51.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:51.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:51.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:51.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:51.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:51.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:51.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:51.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:51.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:51.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:51.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:51.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:51.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:51.593+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:51.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:51.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:51.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:51.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:51.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:51.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:51.609 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1006714013 2023-10-14T00:05:51.616 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:05:51.617 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241951 2023-10-14T00:05:51.617 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:05:51.617 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1006714013 ]] 2023-10-14T00:05:51.618 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1006714013 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\0\6\7\1\4\0\1\3 ]] 2023-10-14T00:05:51.618 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:05:53.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:53.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:53.807 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:53.809+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:53.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:53.817+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:53.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:53.817+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:53.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:53.817+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:54.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:54.245+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:54.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:54.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:54.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:54.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:54.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:54.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:54.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:54.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:05:54.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:54.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:54.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:54.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:54.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:54.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:54.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:54.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:54.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:54.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:55.109 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:55.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:55.109 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:55.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:55.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:55.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:55.130 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:55.129+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:55.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:55.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:55.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:55.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:55.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:55.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:55.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:55.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:55.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:55.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:55.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:55.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:55.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:55.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:55.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:55.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:55.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:55.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:56.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:56.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:56.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:56.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:56.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:56.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:56.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:56.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:56.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:56.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:57.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:57.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:57.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:57.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:57.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:57.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:57.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:57.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:57.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:57.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:57.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:57.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:57.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:57.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:05:57.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:05:57.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:05:57.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:05:57.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:05:57.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:05:57.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:05:57.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:05:59.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.245+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:59.246 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.249+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:05:59.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.249+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:05:59.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.249+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:05:59.569 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.569+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:05:59.569 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.569+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:05:59.591 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.593+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:05:59.591 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.593+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:05:59.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:05:59.593+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:00.020 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.021+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.020 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.021+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8879380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfc880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfd800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfda00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.033+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfdc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.050 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.053+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.053+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfde00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.053+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.053+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.065 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.065+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.065 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.065+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.066 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.065+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.066 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.065+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.081+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.081+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.083 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.085+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.084 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.085+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.085+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.085+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.095 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.097+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.097+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.105+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.105+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d00e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.109+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.109+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d01000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.113+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.113+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d01800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.117+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.117+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d01680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.127 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.129+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.127 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.129+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d01480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.136 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.137+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.137+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d01200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.137+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:06:00.137+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8d01a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:06:00.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:00.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:00.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:00.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:00.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:00.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:00.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:00.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:00.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:00.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:01.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:01.265+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:06:01.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:01.265+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:01.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:01.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:01.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:01.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:01.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:01.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:01.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:01.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:01.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:01.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:01.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:01.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:01.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:01.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:01.624 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:06:01.625 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:01.631 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:01.631 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:02.836 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1006714013 2023-10-14T00:06:02.842 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:06:02.843 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241962 2023-10-14T00:06:02.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:02.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1006714013 ]] 2023-10-14T00:06:02.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1006714013 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\0\6\7\1\4\0\1\3 ]] 2023-10-14T00:06:02.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:06:03.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:03.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:03.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:03.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:03.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:03.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:03.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:03.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:03.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:03.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:03.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:05.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:05.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:05.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:05.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:05.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:05.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:05.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:05.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:05.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:05.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:06.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:06.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:06.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:06.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:06.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:06.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:06.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:06.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:06.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:06.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:08.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:08.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:08.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:08.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:08.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:08.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:08.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:08.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:08.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:08.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:10.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:10.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:10.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:10.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:10.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:10.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:10.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:10.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:10.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:10.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:10.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:10.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:10.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:10.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:11.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:11.389+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:06:11.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:11.389+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:11.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:11.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:11.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:06:11.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:11.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:11.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:11.434 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:11.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:11.434 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:11.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:11.434 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:11.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:11.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:11.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:11.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:11.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:11.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:11.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:11.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:11.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:11.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:11.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:11.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:11.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:11.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:11.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:11.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:11.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:11.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:11.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:11.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:11.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:11.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:11.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:11.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:11.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:11.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:12.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:12.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:12.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:12.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:12.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:12.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:12.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:12.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:12.845 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:06:12.846 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:12.847 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:12.854 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:12.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:12.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:13.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:13.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:13.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:13.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:13.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:13.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:13.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:13.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:13.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:13.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:13.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:13.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:13.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:13.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:13.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:13.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:13.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:14.194 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1006714013 2023-10-14T00:06:14.195 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:06:14.197 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241974 2023-10-14T00:06:14.197 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:14.197 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1006714013 ]] 2023-10-14T00:06:14.198 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1006714013 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\0\6\7\1\4\0\1\3 ]] 2023-10-14T00:06:14.198 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:06:15.314 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:15.313+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:15.349 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:15.349+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:15.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:15.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:15.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:15.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:15.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:15.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:15.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:15.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:15.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:15.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:15.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:15.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:15.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:15.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:15.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:15.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:15.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:15.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:15.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:15.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:16.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:16.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:16.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:16.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:16.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:16.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:16.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:16.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:16.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:16.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:17.064 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:17.065+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:17.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:17.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:17.089 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:17.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:17.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:17.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:17.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:17.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:17.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:17.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:18.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:18.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:18.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:18.797+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:18.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:18.797+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:18.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:18.797+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:18.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:18.797+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:19.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:19.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:19.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:19.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:19.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:19.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:19.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:19.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:19.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:19.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:19.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:19.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:20.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:20.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:20.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:20.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:20.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:20.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:20.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:20.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:20.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:20.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:20.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:20.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:21.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:21.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:21.580 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:21.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:21.593 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:21.593+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:21.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:21.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:21.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:21.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:21.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:21.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:22.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:22.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:22.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:22.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:22.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:22.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:22.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:22.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:22.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:23.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:23.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:23.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:23.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:23.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:23.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:23.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:23.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:23.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:23.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:23.769+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:23.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:23.769+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:23.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:23.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:23.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:23.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:23.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:23.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:23.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:23.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:24.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:24.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:24.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:24.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:24.200 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:24.205 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:24.206 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:06:24.226 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:24.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:24.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:24.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:24.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:24.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:24.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:24.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:24.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:24.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:24.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:24.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:24.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:24.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:24.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:24.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:24.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:24.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:24.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:24.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:06:24.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:24.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:24.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:24.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:24.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:24.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:24.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:24.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:24.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:24.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:06:24.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:24.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:25.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:25.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:25.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:25.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:25.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:25.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:25.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:25.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:25.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:25.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:25.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:25.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:25.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:25.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:25.404 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:25.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:25.404 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:25.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:25.404 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:25.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:25.744 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1006714013 2023-10-14T00:06:25.744 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:06:25.744 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241985 2023-10-14T00:06:25.745 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:25.745 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1006714013 ]] 2023-10-14T00:06:25.745 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1006714013 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\0\6\7\1\4\0\1\3 ]] 2023-10-14T00:06:25.745 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:06:26.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:26.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:26.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:26.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:26.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:26.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:26.648 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:26.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:26.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:26.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:26.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:26.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:26.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:26.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:26.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:26.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:26.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:26.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:26.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:26.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:26.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:27.961 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:27.957+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:27.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:27.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:28.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:06:28.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:28.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.093+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:28.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:28.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:28.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:28.203 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.205+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:06:28.203 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.205+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:28.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:28.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.505+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:28.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:28.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:28.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:28.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:29.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:29.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:29.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:29.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:29.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:29.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:29.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:29.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:29.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:29.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:29.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:29.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:30.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:30.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:30.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:30.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:30.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:30.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:30.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:30.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:30.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:30.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:30.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:30.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:30.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:30.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:30.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:31.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:31.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:31.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:31.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:31.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:31.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:31.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:31.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:31.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:31.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:31.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:31.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:31.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:31.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:33.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:33.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:33.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:33.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:33.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:33.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:34.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:34.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:34.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:34.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:34.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:34.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:34.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:34.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:34.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:34.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:34.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:34.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:34.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:34.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:34.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:34.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:34.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:34.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:34.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:35.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:35.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:35.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:35.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:35.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:35.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:35.109 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:35.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:35.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:35.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:35.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:35.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:35.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:35.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:35.747 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:06:35.779 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:35.780 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:35.780 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:35.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:35.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:35.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:35.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:35.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:35.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:35.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:35.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:35.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:35.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:36.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:36.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:36.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:36.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:36.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:36.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:36.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:36.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:36.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:36.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:36.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:36.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:36.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:36.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:36.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:36.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:36.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:36.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:37.173 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1006714013 2023-10-14T00:06:37.185 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:06:37.186 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697241997 2023-10-14T00:06:37.186 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:37.186 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1006714013 ]] 2023-10-14T00:06:37.187 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1006714013 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\0\6\7\1\4\0\1\3 ]] 2023-10-14T00:06:37.187 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:06:37.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:37.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:37.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:37.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:37.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:37.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:37.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:37.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:38.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:38.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:38.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:38.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:38.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:38.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:38.460 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.460+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:38.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.484+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:38.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.500+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:38.504 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:38.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:38.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:38.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:38.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:38.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:38.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:38.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:38.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:38.908 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:38.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:38.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:38.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:38.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:38.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:38.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:38.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:38.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:39.303 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:39.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:06:39.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:39.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:39.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:39.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:39.311 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:39.312+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:39.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:39.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:39.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:39.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:39.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:39.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:39.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:39.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:39.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:39.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:39.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:39.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:39.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:39.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:39.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:39.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:41.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:41.544+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:41.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:41.560+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:41.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:41.560+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:41.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:41.560+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:42.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:42.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:42.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:42.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:42.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:42.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:42.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:42.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:45.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:45.456+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:45.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:45.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:45.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:45.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:45.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:45.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:45.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:45.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:45.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:45.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:45.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:45.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:45.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:45.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:45.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:45.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:45.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:45.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:45.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:45.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:45.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:45.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:45.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:45.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:45.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:45.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:45.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:45.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:46.125 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:46.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:46.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:46.132+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:46.139 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:46.140+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:46.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:46.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:46.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:46.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:46.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:46.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:47.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:47.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:47.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:47.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:47.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:47.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:47.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:47.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:06:47.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:47.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:47.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:47.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:47.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:47.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:47.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:47.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:47.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:47.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:47.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:47.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:47.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:47.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:47.204 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:47.216 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:47.225 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:06:47.225 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:48.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:48.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:48.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:48.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:48.862 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1862252240 2023-10-14T00:06:48.864 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:06:48.865 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242008 2023-10-14T00:06:48.866 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:48.866 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1862252240 ]] 2023-10-14T00:06:48.866 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1862252240 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\0\0\6\7\1\4\0\1\3 ]] 2023-10-14T00:06:48.867 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/1862252240 2023-10-14T00:06:49.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:06:49.035 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:06:49.035 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:49.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:49.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:49.093 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:49.093 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:49.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:49.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:49.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:49.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:49.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:49.421 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:49.421 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:49.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:50.921 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/1862252240 until 2023-10-14T01:06:50.249024+0000 (3600 sec) 2023-10-14T00:06:50.999 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:06:51.011 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/1862252240 2023-10-14T00:06:51.966 INFO:tasks.workunit.client.0.smithi138.stderr:listed 9 entries 2023-10-14T00:06:51.997 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/1862252240 2023-10-14T00:06:51.997 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:06:52.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:06:52.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:52.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:52.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:52.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:52.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:52.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:52.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:52.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:52.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:54.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:54.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:54.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:54.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:54.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:54.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:54.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:54.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:54.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:54.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:56.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:56.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:56.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:56.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:56.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:56.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:56.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:56.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:06:56.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:56.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:56.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:56.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:56.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:56.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:56.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:56.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:56.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:56.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:56.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:56.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:56.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:56.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:56.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:56.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:56.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:56.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:57.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:57.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:06:57.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:57.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:06:57.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:57.968+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:06:57.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:58.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:06:57.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:58.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:06:57.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:06:58.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:06:59.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:06:59.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:59.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:06:59.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:06:59.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:06:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:59.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:06:59.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:06:59.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:06:59.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:06:59.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:06:59.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:00.003 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.004+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.003 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.004+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae6980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.003 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.004+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.004+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8b60680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.009 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.008+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.008+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d7ab1880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.012+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.012+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8b61080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.016+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.016+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d86ce000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.021 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.020+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.021 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.020+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d896c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.024+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.024+0000 7f082630d640 -1 librbd::ImageState: 0x5627d6295c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b60d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.033 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.032+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.033 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.032+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b60a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.041 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.040+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.040+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ab1300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.064+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.064+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfcd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.064 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.064+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.064 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.064+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b61f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:00.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:00.090 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.092+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.092+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b61d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.093 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:00.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.112+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.112+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfd580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.112+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.113 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.112+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.116+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.116+0000 7f082630d640 -1 librbd::ImageState: 0x5627d6295b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:00.121 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.120+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.122 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.120+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae7780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:00.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:00.134 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.132+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.134 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.132+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b61480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.156 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.156+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.156 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.156+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b60680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.156+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:07:00.156+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b61080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:00.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:00.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:00.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:00.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:00.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:00.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:00.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:00.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:00.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:00.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:00.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:01.594 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:01.592+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:01.617 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:01.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:01.627 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:01.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:01.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:01.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:01.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:01.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:01.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:01.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:01.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:01.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:01.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:01.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:01.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:01.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:01.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:01.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:01.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:01.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:01.998 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:07:01.999 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:01.999 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:02.001 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:03.299 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1862252240 2023-10-14T00:07:03.302 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:07:03.303 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242023 2023-10-14T00:07:03.303 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:03.304 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1862252240 ]] 2023-10-14T00:07:03.304 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1862252240 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\6\2\2\5\2\2\4\0 ]] 2023-10-14T00:07:03.304 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:07:03.694 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:03.696+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:03.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:03.700+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:03.707 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:03.708+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:03.713 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:03.712+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:03.713 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:03.712+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:05.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:05.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:05.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:05.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:05.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:05.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:05.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:05.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:05.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:05.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:05.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:05.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:07.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:07.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:07.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:07.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:07.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:07.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:07.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:07.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:07.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:07.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:07.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:07.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:07.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:07.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:07.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:07.740+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:07:07.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:07.740+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:07.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:07.752+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:07.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:07.752+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:07.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:07.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:07.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:07.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:07.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:07.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:08.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:08.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:08.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:08.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:08.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:08.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:08.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:08.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:08.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:08.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:08.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:08.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:08.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:08.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:09.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:09.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:09.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:09.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:09.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:09.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:09.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:09.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:11.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:11.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:11.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:11.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:11.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:11.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:11.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:11.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:11.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:11.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:12.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:12.136+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:12.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:12.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:12.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:12.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:12.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:12.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:12.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:12.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:13.305 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:07:13.306 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:13.306 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:13.342 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:13.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:13.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:13.736 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:13.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:13.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:13.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:13.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:13.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:13.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:13.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:13.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:13.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:13.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:13.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:13.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:13.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:13.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:14.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:14.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:14.627 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:14.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:14.846 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1862252240 2023-10-14T00:07:14.846 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:07:14.847 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242034 2023-10-14T00:07:14.847 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:14.847 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1862252240 ]] 2023-10-14T00:07:14.847 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1862252240 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\6\2\2\5\2\2\4\0 ]] 2023-10-14T00:07:14.848 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:07:15.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:15.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:15.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:15.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:16.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:16.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:16.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:16.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:16.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:16.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:16.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:16.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:16.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:16.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:16.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:16.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:16.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:16.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:18.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:18.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:18.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:18.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:18.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:18.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:18.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:18.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:18.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:19.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:19.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:19.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:19.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:19.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:19.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:19.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:19.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:19.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:19.589 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:19.588+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:07:19.589 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:19.588+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:19.609 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:19.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:19.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:19.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:19.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:19.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:19.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:19.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:19.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:19.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:20.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:20.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:20.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:20.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:20.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:20.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:20.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:20.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:20.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:21.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:21.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:21.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:21.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:21.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:21.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:21.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:21.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:21.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:21.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:21.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:21.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:21.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:21.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:21.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:21.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:21.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:21.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:21.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:21.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:21.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:21.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:21.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:21.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:22.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:22.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:22.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:22.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:22.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:22.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:22.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:22.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:22.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:22.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:22.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:22.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:23.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:23.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:23.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:23.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:23.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:23.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:23.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:23.347 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.348+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:23.362 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:23.362 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:23.362 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:23.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:24.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:24.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:24.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:24.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:24.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:24.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:24.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:24.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:24.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:24.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:24.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:24.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:24.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:24.850 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:07:24.850 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:24.851 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:24.886 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:25.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:07:25.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:25.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:25.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:25.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:25.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:25.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:25.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:25.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:25.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:26.364 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1862252240 2023-10-14T00:07:26.365 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:07:26.365 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242046 2023-10-14T00:07:26.366 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:26.366 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1862252240 ]] 2023-10-14T00:07:26.366 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1862252240 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\6\2\2\5\2\2\4\0 ]] 2023-10-14T00:07:26.366 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:07:27.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:27.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:27.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:27.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:27.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:27.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:27.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:27.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:27.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:27.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:27.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:27.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:27.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:27.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:28.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:28.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:28.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:28.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:28.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:28.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:28.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:28.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:28.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:28.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:28.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:28.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:28.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:28.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:28.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:29.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:29.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:29.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:29.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:29.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:29.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:29.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:29.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:29.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:29.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:29.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:29.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:30.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:30.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:30.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:30.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:30.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:30.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:30.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:30.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:30.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:30.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:30.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:30.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:30.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:30.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:30.385 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:30.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:07:30.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:30.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:30.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:30.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:30.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:30.416+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:30.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:30.416+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:30.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:30.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:30.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:30.480+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:32.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:32.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:32.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:32.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:32.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:32.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:32.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:32.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:32.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:32.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:32.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:32.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:32.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:32.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:32.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:32.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:32.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:32.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:33.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:33.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:07:33.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:33.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:33.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:33.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:33.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:33.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:33.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:33.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:33.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:33.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:33.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:33.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:34.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:34.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:34.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:34.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:35.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:35.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:35.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:35.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:35.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:35.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:35.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:35.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:35.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:36.392 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:36.394 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:07:36.410 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:36.417 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:37.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:37.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:37.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:37.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:37.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:37.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:37.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:37.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:37.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:37.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:37.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:37.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:37.898 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1862252240 2023-10-14T00:07:37.898 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:07:37.899 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242057 2023-10-14T00:07:37.899 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:37.899 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1862252240 ]] 2023-10-14T00:07:37.899 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1862252240 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\6\2\2\5\2\2\4\0 ]] 2023-10-14T00:07:37.899 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:07:39.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:39.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:39.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:39.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:39.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:39.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:39.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:39.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:39.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:39.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:39.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:39.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:39.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:39.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:40.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:40.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:40.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:40.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:40.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:40.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:40.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:40.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:40.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:42.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:42.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:42.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:42.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:42.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:42.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:42.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:42.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:42.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:42.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:42.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:42.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:43.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.696+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:07:43.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.696+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:43.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.716+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:43.734 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:43.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:43.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:43.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:43.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:43.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:43.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:43.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:44.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:44.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:44.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:44.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:44.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:44.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:44.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:44.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:45.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:45.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:45.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:45.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:45.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:45.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:45.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:45.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:45.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:45.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:46.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:46.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:46.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:46.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:46.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:46.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:47.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:47.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:47.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:47.416+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:47.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:47.428+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:47.427 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:47.428+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:47.427 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:47.428+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:47.901 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:47.906 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:47.909 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:47.916 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:07:48.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:48.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:48.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:48.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:48.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:48.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:48.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:48.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:48.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:48.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:48.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:48.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:48.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:48.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:49.227 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1862252240 2023-10-14T00:07:49.230 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:07:49.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242069 2023-10-14T00:07:49.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:49.231 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1862252240 ]] 2023-10-14T00:07:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1862252240 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\6\2\2\5\2\2\4\0 ]] 2023-10-14T00:07:49.233 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:07:49.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:49.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:49.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:49.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:49.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:49.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:49.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:49.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:49.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:49.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:49.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:49.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:49.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:49.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:51.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:51.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:51.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:51.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:51.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:51.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:51.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:51.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:51.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:51.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:51.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:52.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:52.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:07:52.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:52.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:52.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:52.440+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:52.447 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:52.448+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:52.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:52.452+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:53.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:07:53.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:07:53.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:53.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:53.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:53.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:53.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:53.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:53.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:53.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:53.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:53.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:53.704+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:53.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:53.720+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:53.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:53.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:53.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:53.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:53.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:53.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:53.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:53.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:54.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:54.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:07:54.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:54.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:55.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:55.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:55.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:55.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:55.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:55.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:55.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:55.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:55.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:55.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:55.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:55.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:55.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:55.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:55.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:55.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:55.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:55.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:55.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:55.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:56.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:56.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:56.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:56.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:56.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:56.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:57.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:57.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:57.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:57.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:57.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:57.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:57.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:57.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:57.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:57.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:07:57.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:57.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:57.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:57.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:57.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:57.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:57.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:57.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:57.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:57.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:57.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:57.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:57.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:57.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:58.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:07:58.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:07:58.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:58.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:07:58.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:58.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:58.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:58.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:58.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:58.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:58.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:07:58.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:07:58.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:07:58.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:07:58.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:07:58.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:07:58.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:07:58.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:07:59.235 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:59.241 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:07:59.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:07:59.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:07:59.247 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:59.257 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:59.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:07:59.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:07:59.259 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:07:59.259 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:07:59.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:00.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:07:59.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:00.014 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:00.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:00.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:00.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:00.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:00.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:00.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:00.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:00.645 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3331146725 2023-10-14T00:08:00.645 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:08:00.649 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242080 2023-10-14T00:08:00.649 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:00.649 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3331146725 ]] 2023-10-14T00:08:00.649 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3331146725 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\6\2\2\5\2\2\4\0 ]] 2023-10-14T00:08:00.650 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3331146725 2023-10-14T00:08:01.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:01.875+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:01.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:01.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:01.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:01.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:01.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:01.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:01.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:01.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:01.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:01.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:02.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:02.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:02.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:02.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:02.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:02.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:02.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:02.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:02.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:02.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:03.253 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3331146725 until 2023-10-14T01:08:02.620246+0000 (3600 sec) 2023-10-14T00:08:03.317 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:08:03.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3331146725 2023-10-14T00:08:03.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:03.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:03.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:03.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:03.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:03.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:03.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:03.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:03.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:03.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:04.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:03.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:04.471 INFO:tasks.workunit.client.0.smithi138.stderr:listed 10 entries 2023-10-14T00:08:04.518 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3331146725 2023-10-14T00:08:04.519 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:08:04.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:04.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:04.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:04.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:04.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:04.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:05.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:05.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:05.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:05.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:05.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:05.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:05.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:05.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:05.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:05.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:05.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:05.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:06.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:08:06.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:06.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:06.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:06.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:06.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:06.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:06.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:06.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:08.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:08.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:08.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:08.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:08.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:08.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:08.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:08.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:08.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:08.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:08.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:08.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:08.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:08.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:08.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:08.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:08.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:08.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:09.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:09.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:09.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:09.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:09.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:09.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:09.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:09.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:09.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:09.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:09.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:09.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:10.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:10.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:10.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:10.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:10.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:10.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:10.559 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:10.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:10.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:10.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:10.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:10.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:10.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:10.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:14.526 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:14.541 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:14.548 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:08:14.554 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:15.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:15.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:15.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:15.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:15.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:15.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:15.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:15.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:15.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:15.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:15.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:15.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:15.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:15.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:15.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:15.795+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:08:15.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:15.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:15.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:15.819+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:15.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:15.819+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:15.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:15.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:15.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:15.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:15.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:15.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:16.072 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3331146725 2023-10-14T00:08:16.072 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:08:16.073 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242096 2023-10-14T00:08:16.073 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:16.073 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3331146725 ]] 2023-10-14T00:08:16.073 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3331146725 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\3\3\1\1\4\6\7\2\5 ]] 2023-10-14T00:08:16.073 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:08:16.755 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:16.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:16.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:16.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:16.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:16.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:16.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:16.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:18.235 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:08:18.235+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:08:19.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:19.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:19.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:19.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:19.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:19.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:19.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:19.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:19.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:19.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:19.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:19.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:21.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:21.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:21.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:21.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:21.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:21.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:21.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:21.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:21.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:21.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:21.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:23.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:23.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:23.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:23.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:23.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:23.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:23.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:25.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:25.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:25.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:25.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:25.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:25.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:25.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:25.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:25.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:25.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:25.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:25.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:26.076 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:08:26.076 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:26.080 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:26.085 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:26.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:26.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:26.525 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:26.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:26.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:26.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:26.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:26.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:27.151 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3331146725 2023-10-14T00:08:27.151 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:08:27.154 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242107 2023-10-14T00:08:27.154 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:27.155 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3331146725 ]] 2023-10-14T00:08:27.155 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3331146725 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\3\3\1\1\4\6\7\2\5 ]] 2023-10-14T00:08:27.155 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:08:27.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:27.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:27.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:27.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:27.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:27.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:27.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:27.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:27.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:27.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:27.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:27.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:27.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:27.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:27.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:27.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:27.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:27.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:27.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:27.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:27.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:27.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:27.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:27.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:27.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:27.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:28.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:27.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:28.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:28.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:28.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:28.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:28.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:28.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:28.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:28.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:28.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:28.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:28.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:28.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:28.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:28.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:28.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:28.551+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:28.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:28.551+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:28.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:28.555+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:28.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:28.587+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:28.648 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:28.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:28.648 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:28.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:29.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:29.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:29.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:29.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:29.183 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:29.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:29.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:29.191+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:29.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:29.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:29.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:29.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:29.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:29.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:29.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:29.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:29.678 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:08:29.679+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:08:29.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:29.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:29.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:29.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:29.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:29.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:29.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:34.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:34.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:34.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:34.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:34.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:34.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:34.015 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:34.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:34.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:34.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:34.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:34.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:34.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:34.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:34.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:34.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:34.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:34.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:34.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:34.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:34.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:34.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:34.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:34.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:34.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:34.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:35.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:35.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:35.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:35.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:35.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:35.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:36.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:08:36.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:36.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:36.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:36.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:36.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:36.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:36.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:36.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:36.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:36.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:36.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:36.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:36.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:36.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:36.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:36.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:36.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:37.160 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:37.169 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:08:37.170 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:37.172 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:38.199 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.199+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:08:38.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:38.207 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:38.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:38.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.239+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:38.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:38.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:38.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:38.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:38.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:38.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:38.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:38.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:38.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:38.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:38.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:38.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:38.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:38.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:38.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:38.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:38.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:38.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:38.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:38.696 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3331146725 2023-10-14T00:08:38.696 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:08:38.696 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242118 2023-10-14T00:08:38.697 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:38.697 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3331146725 ]] 2023-10-14T00:08:38.697 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3331146725 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\3\3\1\1\4\6\7\2\5 ]] 2023-10-14T00:08:38.697 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:08:39.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:39.867+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:39.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:39.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:39.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:39.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:39.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:39.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:39.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:39.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:41.150 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:41.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:41.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:41.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:41.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:41.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:41.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:41.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:41.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:41.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:41.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:41.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:41.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:41.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:41.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:41.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:41.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:41.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:41.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:41.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:41.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:41.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:41.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:41.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:43.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:43.839+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:43.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:43.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:43.857 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:43.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:43.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:43.867+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:43.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:43.867+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:43.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:43.867+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:44.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:44.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:44.064 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:44.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:44.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:44.079+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:44.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:44.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:44.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:44.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:44.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:44.091+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:44.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:44.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:44.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:44.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:44.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:08:44.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:08:44.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:44.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:44.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:44.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:44.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:44.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:44.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:44.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:44.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:45.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:45.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:45.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:45.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:45.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:45.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:45.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:08:45.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:45.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:45.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:45.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:45.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:45.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:45.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:45.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:45.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:45.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:08:45.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:45.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:45.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:45.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:45.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:45.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:45.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:45.699+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:45.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:45.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:45.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:45.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:45.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:45.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:45.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:45.795+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:45.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:45.795+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:46.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.179+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:08:46.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.179+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:08:46.212 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:46.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:46.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:46.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:46.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:46.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:46.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:46.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:46.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:46.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:46.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:46.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:46.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:46.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:46.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:46.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.683+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:46.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.699+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:08:46.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:46.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:46.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:46.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:08:46.859 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:08:46.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:08:46.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:08:46.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:08:47.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:47.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:47.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:47.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:47.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:47.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:47.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:47.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:48.706 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:48.707 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:48.714 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:08:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:49.955 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3331146725 2023-10-14T00:08:49.956 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:08:49.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242129 2023-10-14T00:08:49.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:49.957 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3331146725 ]] 2023-10-14T00:08:49.957 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3331146725 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\3\3\1\1\4\6\7\2\5 ]] 2023-10-14T00:08:49.957 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:08:53.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:53.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:08:53.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:53.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:53.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:53.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:08:53.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:53.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:53.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:53.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:53.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:53.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:53.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:53.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:55.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:55.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:55.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:55.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:56.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:56.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:08:56.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:56.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:56.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:56.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:57.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:57.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:08:57.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:57.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:58.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:58.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:08:58.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:58.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:08:58.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:08:58.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:08:59.966 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:59.967 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:59.993 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:09:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.047+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.047+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.055+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.055+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.063+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.063+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.064 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.063+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.064 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.063+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.065 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.063+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.065 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.063+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.095+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.095+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.095+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.095+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b19100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.119+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.119+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.173 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.175+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.173 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.175+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.174 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.175+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b19680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.174 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.175+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b19500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.179+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.179+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b19300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.194 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.195+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.194 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.195+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b18d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.221 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.221 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b19900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.222 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.222 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.223 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b19b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b19d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b19f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:00.219+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd6700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:01.514 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3331146725 2023-10-14T00:09:01.515 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:09:01.517 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242141 2023-10-14T00:09:01.518 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:01.518 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3331146725 ]] 2023-10-14T00:09:01.518 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3331146725 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\3\3\1\1\4\6\7\2\5 ]] 2023-10-14T00:09:01.518 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:09:01.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:01.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:01.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:01.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:01.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:01.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:01.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:01.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:01.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:01.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:01.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:01.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:01.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:01.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:02.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:02.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:02.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:02.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:02.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:02.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:02.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:02.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:02.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:02.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:02.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:02.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:03.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:03.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:03.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:03.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:03.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:03.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:03.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:03.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:03.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:03.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:03.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:04.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:04.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:04.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:04.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:04.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:04.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:04.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:04.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:04.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:04.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:04.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:04.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:04.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:04.903+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:04.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:04.903+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:04.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:04.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:04.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:04.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:04.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:04.915+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:04.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:04.915+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:04.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:04.915+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:05.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:05.251+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:05.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:05.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:05.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:05.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:05.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:05.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:05.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:05.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:08.125 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.127+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:08.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.135+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:08.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:08.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:08.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:08.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.507+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:08.509 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.507+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:08.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:08.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:08.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:08.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:08.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:08.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:09.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:09.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:09.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:09.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:09.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:09.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:09.382 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:09.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:09.382 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:09.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:09.383 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:09.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:10.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:10.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:10.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:10.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:11.521 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:09:11.529 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:11.530 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:11.559 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:11.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:11.811+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:11.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:11.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:11.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:11.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:11.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:11.863+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:11.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:11.863+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:11.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:11.863+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:13.223 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/359867429 2023-10-14T00:09:13.224 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:09:13.224 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242153 2023-10-14T00:09:13.224 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:13.225 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/359867429 ]] 2023-10-14T00:09:13.225 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/359867429 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\3\3\1\1\4\6\7\2\5 ]] 2023-10-14T00:09:13.225 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/359867429 2023-10-14T00:09:14.331 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.331+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:14.331 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.331+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:14.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:14.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:14.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:14.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:14.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:14.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:14.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:14.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:14.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:14.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:14.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:14.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:14.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:14.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:14.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:14.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:14.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:14.993 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:14.995+0000 7f082630d640 -1 librbd::image::GetMetadataRequest: 0x5627d364c480 handle_metadata_list: failed to retrieve image metadata: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:14.993 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:14.995+0000 7f082630d640 -1 librbd::image::RefreshRequest: failed to retrieve pool metadata: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:14.993 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:14.995+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to refresh image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:14.994 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:14.995+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8bfd480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:09:15.973 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/359867429 until 2023-10-14T01:09:15.353884+0000 (3600 sec) 2023-10-14T00:09:16.031 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/359867429 2023-10-14T00:09:16.038 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:09:17.027 INFO:tasks.workunit.client.0.smithi138.stderr:listed 11 entries 2023-10-14T00:09:17.072 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/359867429 2023-10-14T00:09:17.073 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:09:17.110 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:17.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:17.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:17.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:17.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:17.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:17.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:17.179+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:17.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:17.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:17.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:17.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:17.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:17.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:17.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:17.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:17.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:17.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:17.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:17.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:17.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:17.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:17.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:18.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:18.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:18.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:18.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:18.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:18.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:18.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:18.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:18.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:18.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:18.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:18.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:18.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:19.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:19.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:19.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:19.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:19.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:19.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:19.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:19.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:19.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:19.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:20.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:09:20.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:20.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:20.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:20.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:20.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:20.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:20.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:20.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:21.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:21.551+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:21.589 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:21.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:21.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:21.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:21.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:21.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:21.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:21.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:22.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:22.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:22.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:22.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:22.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:22.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:23.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:23.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:23.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:23.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:23.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:23.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:23.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:23.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:23.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:23.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:23.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:23.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:23.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:23.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:24.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:24.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:24.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:24.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:24.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:24.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:24.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:24.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:25.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:09:25.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:25.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:25.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:25.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:25.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:25.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:25.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:25.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:25.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:25.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:25.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:25.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:25.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:25.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:27.081 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:27.085 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:27.088 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:09:27.106 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:27.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:27.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:27.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:27.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:27.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:27.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:27.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:27.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:27.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:27.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:27.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:28.460 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2888421112 2023-10-14T00:09:28.460 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:09:28.461 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242168 2023-10-14T00:09:28.461 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:28.461 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2888421112 ]] 2023-10-14T00:09:28.461 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2888421112 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\5\9\8\6\7\4\2\9 ]] 2023-10-14T00:09:28.461 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2888421112 2023-10-14T00:09:29.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:29.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:29.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:29.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:29.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:29.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:09:29.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:09:29.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:29.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:29.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:29.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:29.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:30.894 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2888421112 until 2023-10-14T01:09:30.138609+0000 (3600 sec) 2023-10-14T00:09:30.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:09:30.957 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2888421112 2023-10-14T00:09:31.009 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:31.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:31.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:31.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:31.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:31.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:31.048 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:31.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:31.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:31.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:31.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:31.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:31.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:31.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:31.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:31.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:31.918 INFO:tasks.workunit.client.0.smithi138.stderr:listed 12 entries 2023-10-14T00:09:31.965 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2888421112 2023-10-14T00:09:31.966 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:09:32.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:32.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:33.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:33.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:33.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:33.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:33.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:33.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:33.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:33.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:33.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:33.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:33.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:33.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:35.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:35.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:35.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:35.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:35.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:35.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:35.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:35.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:35.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:35.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:35.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:35.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:35.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:35.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:36.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:36.770+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:36.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:36.770+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:36.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:36.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:36.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:36.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:36.802 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:36.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:36.802 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:36.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:36.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:36.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:37.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:37.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:38.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:38.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:39.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:39.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:39.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:39.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:39.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:39.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:39.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:39.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:39.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:39.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:39.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:39.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:40.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:40.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:40.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:40.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:40.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:40.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:40.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:40.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:40.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:40.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:40.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:40.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:40.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:40.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:41.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:41.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:41.706 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:41.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:41.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:41.742+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:41.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:41.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:41.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:41.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:41.980 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:09:41.981 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:41.981 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:41.992 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:43.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:43.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:43.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:43.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:43.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:43.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:43.149 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:43.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:43.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:43.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:43.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:43.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:43.335 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2888421112 2023-10-14T00:09:43.341 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:09:43.342 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242183 2023-10-14T00:09:43.342 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:43.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2888421112 ]] 2023-10-14T00:09:43.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2888421112 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\8\8\4\2\1\1\1\2 ]] 2023-10-14T00:09:43.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:09:44.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:44.970+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:44.986 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:44.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:44.993 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:44.994+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:45.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:45.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:45.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:45.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:46.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:46.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.122+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:46.139 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.138+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:46.454 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:46.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:46.487 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.486+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:46.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:46.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:46.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:46.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:46.936 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.934+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:46.936 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.934+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:46.936 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:46.934+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:48.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:48.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:48.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:48.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:48.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:48.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:48.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:48.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:48.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:50.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:50.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:50.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:50.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:50.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:50.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:50.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:50.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:50.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:50.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:50.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:50.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:50.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:50.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:51.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:51.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:51.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:51.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:51.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:51.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:51.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:51.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:51.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:51.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:52.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:52.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:52.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:52.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:52.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:52.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:52.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:52.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:52.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:52.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:52.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:52.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:52.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:52.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:53.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:53.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:53.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:53.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:53.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:53.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:53.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:53.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:53.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:53.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:53.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:53.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:53.345 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:53.349 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:53.355 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:53.356 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:09:53.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:53.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:53.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:53.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:53.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:53.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:54.853 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2888421112 2023-10-14T00:09:54.853 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:09:54.854 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242194 2023-10-14T00:09:54.854 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:54.854 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2888421112 ]] 2023-10-14T00:09:54.855 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2888421112 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\8\8\4\2\1\1\1\2 ]] 2023-10-14T00:09:54.855 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:09:55.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:55.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:55.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:55.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:55.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:55.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:55.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:55.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:55.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:55.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:09:55.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:09:55.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:55.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:09:55.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:09:55.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:09:55.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:55.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:55.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:55.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:09:58.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:58.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:09:58.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:58.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:09:58.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:58.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:09:58.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:58.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:09:58.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:58.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:09:58.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:58.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:09:58.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:09:58.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:09:59.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:59.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:09:59.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:59.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:09:59.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:59.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:09:59.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:09:59.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:00.000 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:59.998+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.001 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:09:59.998+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aae580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.006 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.006+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.006+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8282d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.018 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.018+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.018+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b19600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.033 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.034+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.034 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.034+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d86c8a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.035 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.034+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.035 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.034+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.060 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.058+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.060 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.058+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b60e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.068 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.070+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.070+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a8900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.070+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.070+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d7cc2200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.088 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.086+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.088 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.086+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd7e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.092 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.094+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.094+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd7380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.094+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.094+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8879080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.094+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.094+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d81a9d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.101 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.102+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.102+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd7c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.106+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.106+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd7a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.106+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.106+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd7800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.114+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.116 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.114+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd7600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.154 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.154+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.154 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.154+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8860000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.155 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.154+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.155 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.154+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8860200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.175 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.174+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.176 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.174+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8860400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.205 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.206+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.205 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:10:00.206+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8860600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:00.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:00.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:00.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:00.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:00.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:10:00.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:00.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:00.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:00.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:00.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:00.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:00.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:01.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:01.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:01.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:01.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:01.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:01.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:01.102 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:01.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:01.102 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:01.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:02.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:02.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:02.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:02.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:02.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:02.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:02.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:02.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:03.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:03.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:03.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:03.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:03.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:03.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:03.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:03.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:03.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:10:03.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:03.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:03.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:03.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:03.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:03.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:03.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:03.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:03.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:03.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:03.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:03.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:03.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:03.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:03.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:03.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:03.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:03.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:03.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:04.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:04.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:04.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:04.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:04.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:04.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:04.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:04.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:04.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:04.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:04.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:04.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:04.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:04.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:04.866 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:04.879 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:10:05.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:05.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:05.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:05.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:05.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:05.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:05.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:05.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:05.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:05.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:05.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:05.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:05.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:05.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:06.327 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2888421112 2023-10-14T00:10:06.328 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:10:06.329 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242206 2023-10-14T00:10:06.329 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:06.329 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2888421112 ]] 2023-10-14T00:10:06.330 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2888421112 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\8\8\4\2\1\1\1\2 ]] 2023-10-14T00:10:06.330 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:10:07.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:07.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:07.149 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:07.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:07.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:07.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:07.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:07.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:07.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:07.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:07.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:07.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:07.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:07.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:08.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:08.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:08.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:08.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:08.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:08.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:09.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:09.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:09.194 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:09.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:09.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:09.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:09.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:09.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:09.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:09.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:10.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:10.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:10.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:10.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:10.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:10.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:10.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:10.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:10.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:10.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:11.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:10:11.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:11.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:11.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:11.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:11.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:11.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:11.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:11.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:11.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:11.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:10:11.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:11.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:11.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:11.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:11.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:11.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:11.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:11.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:11.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:11.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:11.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.618+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:11.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:11.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:11.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:11.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:11.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:11.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:11.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:11.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:14.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:14.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.566+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:14.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:14.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:14.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:14.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:14.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.934+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:14.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:14.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:14.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:14.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:14.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:14.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:14.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:14.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:14.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:14.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:14.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:15.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:15.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:15.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:15.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:15.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:15.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:15.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:15.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:15.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:15.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:15.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:15.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:15.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:15.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:15.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:15.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:15.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:15.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:15.064 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:15.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:15.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:15.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:15.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:15.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:15.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:15.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:16.330 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:10:16.331 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:16.332 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2888421112 2023-10-14T00:10:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:10:17.509 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242217 2023-10-14T00:10:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2888421112 ]] 2023-10-14T00:10:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2888421112 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\8\8\4\2\1\1\1\2 ]] 2023-10-14T00:10:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:10:17.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:17.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:17.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:17.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:17.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:17.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:17.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:17.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:17.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:23.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:23.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:23.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:23.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:23.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:23.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:23.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:23.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:23.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:23.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:25.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:25.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:25.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:25.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:25.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:25.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:25.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:25.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:25.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:26.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:26.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:26.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:26.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:26.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:26.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:26.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:26.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:26.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:26.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:27.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:27.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:27.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:27.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:27.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:27.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:27.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:27.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:27.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:27.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:27.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:27.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:27.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:27.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:27.512 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:27.513 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:27.513 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:27.531 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:10:28.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:28.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:28.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:28.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:29.066 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2888421112 2023-10-14T00:10:29.066 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:10:29.068 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242229 2023-10-14T00:10:29.068 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:29.068 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2888421112 ]] 2023-10-14T00:10:29.068 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2888421112 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\8\8\4\2\1\1\1\2 ]] 2023-10-14T00:10:29.068 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:10:29.699 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:29.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:29.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:29.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:29.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:29.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:29.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:29.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:29.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:29.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:30.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:30.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:30.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:30.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:30.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:30.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:30.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:30.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:30.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:30.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:30.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:30.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:30.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:30.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:31.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:31.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:31.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:31.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:31.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:31.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:31.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:31.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:31.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:31.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:31.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:31.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:31.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:32.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:32.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:33.707 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:33.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:33.707 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:33.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:33.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:33.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:33.741 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:33.742+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:33.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:33.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:33.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:33.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:33.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:33.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:34.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:34.518+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:34.535 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:34.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:34.561 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:34.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:34.587 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:34.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:34.587 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:34.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:34.587 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:34.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:35.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:35.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:35.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:35.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:35.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:35.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:35.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:35.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:35.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:35.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:35.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:35.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:35.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:36.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:10:36.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:36.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:36.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:36.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:36.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:36.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:36.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:36.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:36.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:36.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:37.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:37.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:37.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:37.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:37.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:37.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:37.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:37.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:38.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:38.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:38.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:38.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:38.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:38.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:38.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:38.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:38.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:38.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:38.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:38.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:38.838+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:39.079 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:10:39.080 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:39.094 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:39.273 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:39.274+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:39.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:39.274+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:39.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:39.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:39.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:39.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:39.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:39.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:39.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:39.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:39.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:39.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:39.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:39.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:39.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:39.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:39.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:39.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:39.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:39.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:39.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:39.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:40.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:40.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:40.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:40.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:40.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:40.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:40.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:40.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:40.739 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3864440564 2023-10-14T00:10:40.740 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:10:40.740 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242240 2023-10-14T00:10:40.741 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:40.741 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3864440564 ]] 2023-10-14T00:10:40.741 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3864440564 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\8\8\4\2\1\1\1\2 ]] 2023-10-14T00:10:40.741 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3864440564 2023-10-14T00:10:41.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:41.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:41.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:41.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:41.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:41.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:41.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:41.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:41.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:41.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:41.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:41.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:41.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:41.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:41.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:41.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:41.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:41.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:10:41.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:41.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:41.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:41.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:41.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:41.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:41.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:41.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:41.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:41.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:42.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:42.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:42.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:42.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:42.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:42.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:42.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:42.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:42.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:42.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:42.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:42.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:42.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:42.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:43.000 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3864440564 until 2023-10-14T01:10:42.280773+0000 (3600 sec) 2023-10-14T00:10:43.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:43.047 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:10:43.052 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3864440564 2023-10-14T00:10:43.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:43.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:43.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:43.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:43.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:43.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:43.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:43.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:43.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:43.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:43.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:43.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:44.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:44.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:44.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:44.280 INFO:tasks.workunit.client.0.smithi138.stderr:listed 13 entries 2023-10-14T00:10:44.326 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3864440564 2023-10-14T00:10:44.326 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:10:44.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:44.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:44.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:44.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:44.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:44.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:44.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:44.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:46.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:46.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:46.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:46.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:46.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:46.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:46.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:46.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:46.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:46.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:46.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:46.154+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:46.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:46.154+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:46.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:46.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:46.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:46.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:46.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:46.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:46.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:46.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:46.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:46.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:46.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:46.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:47.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:47.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:47.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:47.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:47.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:47.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:47.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:47.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:47.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:47.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:47.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:47.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:47.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:47.186+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:47.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:47.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:47.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:47.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:47.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:47.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:49.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:49.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:49.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:49.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:49.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:49.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:49.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:49.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:49.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:49.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:49.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:49.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:49.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:49.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:50.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:50.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:50.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:50.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:50.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:50.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:50.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:50.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:50.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:50.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:50.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:50.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:50.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:50.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:51.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:51.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:51.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:51.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:51.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:51.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:10:51.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:51.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:51.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:51.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:51.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:51.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:51.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:51.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:51.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:51.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:52.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.026+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:10:52.035 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:52.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:52.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:52.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:52.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:52.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:52.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:52.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:53.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:53.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:53.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:53.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:53.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:53.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:53.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:53.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:53.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:53.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:53.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:53.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:53.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:53.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:53.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:53.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:53.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:53.854+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:53.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:53.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:53.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:53.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:53.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:53.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:54.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:54.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:54.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:54.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:54.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:54.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:54.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:54.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:10:54.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:54.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:54.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:54.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:54.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:54.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:54.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:54.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:54.368 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:10:54.369 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:54.371 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:54.398 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:55.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:55.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:55.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:55.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:55.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:55.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:55.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:55.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:55.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:55.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:55.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:55.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:55.958 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:55.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:55.958 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:55.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:55.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:55.978+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:55.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:55.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:55.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:55.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:55.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:55.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:55.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:55.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:56.105 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3864440564 2023-10-14T00:10:56.105 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:10:56.106 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242256 2023-10-14T00:10:56.106 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:56.107 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3864440564 ]] 2023-10-14T00:10:56.107 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3864440564 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\4\4\4\0\5\6\4 ]] 2023-10-14T00:10:56.107 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:10:57.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:57.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:57.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:57.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:57.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:57.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:57.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:57.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:57.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:57.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:57.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:57.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:58.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:58.322+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:10:58.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:58.322+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:10:58.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:58.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:10:58.367 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:58.366+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:10:58.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:58.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:10:58.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:58.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:10:58.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:10:58.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:10:58.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:58.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:10:58.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:58.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:58.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:58.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:58.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:58.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:58.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:58.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:58.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:58.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:58.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:58.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:10:59.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:59.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:10:59.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:59.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:10:59.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:59.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:10:59.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:59.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:10:59.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:59.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:10:59.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:59.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:10:59.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:10:59.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.010+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.010+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.010+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d896c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.010+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae7780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.054+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.058 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.054+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8bfc580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.070+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.070+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.070+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.070+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d7ae6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.070+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8bfd200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.070+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8879300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.074+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.074+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.074+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d81a8480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.074+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d81a8080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.106 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8283600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8283700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8282880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d81a9c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8860980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.113 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8861800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:00.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:00.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:00.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:00.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:00.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:00.161 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.162+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.161 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.162+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8282e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.171 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.170+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.171 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.170+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8861a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:00.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:00.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:00.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:00.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:00.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:00.210 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.210+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.211 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.210+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8861c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.222+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.222+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.226 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.222+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.226 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.222+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8861e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.226 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.222+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dea080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.226 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:11:00.222+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dea280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:01.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:01.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:01.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:01.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:01.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:01.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:01.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:01.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:01.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:01.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:01.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:02.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:02.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:02.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:02.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:02.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:02.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:02.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:02.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:02.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:02.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:05.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:05.237+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:05.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:05.237+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:05.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:05.237+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:05.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:05.237+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:06.113 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:06.115 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:11:06.119 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:06.127 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:07.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:11:07.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:07.303 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3864440564 2023-10-14T00:11:07.303 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:11:07.304 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242267 2023-10-14T00:11:07.304 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:07.305 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3864440564 ]] 2023-10-14T00:11:07.305 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3864440564 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\4\4\4\0\5\6\4 ]] 2023-10-14T00:11:07.305 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:11:07.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:07.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:07.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:07.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:07.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:07.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:07.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:07.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:07.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:07.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:07.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:07.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:07.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:07.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:07.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:07.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:07.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:07.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:07.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:07.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:07.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:07.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:07.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:07.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:07.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:08.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.041+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:08.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:08.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:08.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:08.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:08.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:08.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:08.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:08.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:08.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:08.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:08.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:08.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:08.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:08.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:08.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:08.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:08.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:08.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:08.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:08.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:08.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:08.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:08.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:08.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:08.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:09.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:09.161+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:09.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:09.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:09.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:09.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:10.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:10.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:10.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:10.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:10.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:10.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:10.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:10.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:10.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:10.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:10.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:10.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:10.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:13.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:13.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:13.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:13.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:13.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:13.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:13.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:13.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:13.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:13.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:13.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:13.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:14.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:14.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:11:14.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:14.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:14.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:14.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:14.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:14.301+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:14.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:14.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:14.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:14.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:14.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:14.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:14.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:14.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:14.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:14.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:14.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:14.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:14.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:14.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:14.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:15.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:15.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:15.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:15.957+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:16.002 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:16.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:16.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:16.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:16.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:16.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:16.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:16.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:16.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:16.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:16.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:16.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:16.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:16.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:16.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:16.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:16.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:16.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:16.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:16.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:16.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:16.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:16.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:17.308 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:17.310 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:17.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:17.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:17.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:17.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:17.333 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:17.335 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:11:17.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:17.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:17.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:17.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:17.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:17.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:17.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:17.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:17.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:17.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:18.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:18.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:18.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:18.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:18.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:18.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:18.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:18.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:18.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:18.847 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3864440564 2023-10-14T00:11:18.847 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:11:18.867 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242278 2023-10-14T00:11:18.867 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:18.867 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3864440564 ]] 2023-10-14T00:11:18.867 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3864440564 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\4\4\4\0\5\6\4 ]] 2023-10-14T00:11:18.868 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:11:19.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:19.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:19.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:19.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:19.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:19.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:19.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:19.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:19.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:19.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:19.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:19.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:19.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:19.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:19.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:19.628 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:19.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:19.649 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:19.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:19.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:19.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:19.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:19.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:19.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:19.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:21.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:21.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:21.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:21.865+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:21.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:21.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:21.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:21.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:21.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:21.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:22.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:22.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:22.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:22.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:22.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:22.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:22.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:22.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:22.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:22.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:22.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:22.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:22.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:22.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:23.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:23.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:23.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:23.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:23.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:23.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:23.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:23.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:23.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:23.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:25.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:25.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:25.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:25.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:25.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:25.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:25.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:25.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:25.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:26.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:26.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:26.313 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.313+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:26.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:26.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:26.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:26.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:26.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:26.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:26.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:26.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:11:26.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.689+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:26.696 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:26.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:27.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:27.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:27.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:27.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:27.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:27.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:27.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:27.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:27.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:27.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:27.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:27.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:27.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:27.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:28.883 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:28.883 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:11:28.893 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:28.912 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:30.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:30.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:11:30.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:30.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:30.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:30.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:30.229 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3864440564 2023-10-14T00:11:30.231 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:11:30.288 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242290 2023-10-14T00:11:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3864440564 ]] 2023-10-14T00:11:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3864440564 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\4\4\4\0\5\6\4 ]] 2023-10-14T00:11:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:11:30.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:30.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:30.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:30.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:30.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:30.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:30.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:30.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:30.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:30.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:30.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:30.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:30.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:30.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:30.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:30.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:30.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:30.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:30.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:30.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:30.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:30.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:32.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:32.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:32.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:32.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:32.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:32.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:32.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:32.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:32.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:32.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:32.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:32.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:32.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:32.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:32.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:33.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:33.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:33.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:33.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:33.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:33.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:33.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:33.073+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:33.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:33.073+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:33.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:33.073+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:33.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:33.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:33.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:33.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:33.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:33.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:33.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:33.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:33.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:33.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:33.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:33.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:34.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:34.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:34.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:34.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:34.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:34.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:34.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:34.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:34.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:34.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:34.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:34.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:34.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:34.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:35.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:35.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:35.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:35.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:35.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:35.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:35.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:35.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:35.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:35.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:35.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:35.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:35.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:35.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:35.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:35.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:35.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:35.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:37.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:37.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:11:37.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:37.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:37.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:37.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:37.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:37.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:37.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:37.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:37.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:37.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:38.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:38.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:38.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:38.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:38.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:38.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:38.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:38.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:38.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:38.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:39.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:39.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:39.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:39.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:39.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:39.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:39.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:39.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:39.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:39.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:40.530 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:11:40.531 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:40.531 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:40.532 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:41.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:41.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:41.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:41.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:41.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:41.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:41.928 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3864440564 2023-10-14T00:11:41.931 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:11:42.193 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242302 2023-10-14T00:11:42.193 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:42.193 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3864440564 ]] 2023-10-14T00:11:42.194 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3864440564 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\4\4\4\0\5\6\4 ]] 2023-10-14T00:11:42.194 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:11:44.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:44.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:44.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:44.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:44.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:44.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:44.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:44.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:44.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:44.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:44.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:44.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:44.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:44.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:45.288 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:45.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:11:45.288 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:45.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:45.288 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:45.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:45.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:45.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:45.290 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:45.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:45.290 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:45.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:45.290 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:45.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:45.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:45.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:45.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:45.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:45.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:45.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:45.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:45.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:45.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:45.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:45.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:47.869 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:47.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:47.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:47.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:47.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:47.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:47.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:47.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:49.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:49.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:49.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:49.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:49.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:49.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:49.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:49.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:49.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:49.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:50.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:50.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:50.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:50.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:50.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:50.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:51.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:51.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:51.085 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:51.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:51.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:51.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:51.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:51.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:51.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:51.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:51.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:51.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:51.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:51.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:51.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:51.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:51.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:51.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:51.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:52.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:52.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:52.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:52.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:52.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:52.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:52.214 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:11:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:52.226 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:53.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:11:53.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.088+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:53.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:53.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:53.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.096+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:53.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.096+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:53.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.096+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:53.560 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/947637180 2023-10-14T00:11:53.562 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:11:53.565 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242313 2023-10-14T00:11:53.566 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:53.566 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/947637180 ]] 2023-10-14T00:11:53.566 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/947637180 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\4\4\4\0\5\6\4 ]] 2023-10-14T00:11:53.566 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/947637180 2023-10-14T00:11:53.840 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.840+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:53.869 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:53.872+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:54.102 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:54.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:54.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:54.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:54.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:54.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:54.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:54.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:56.033 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/947637180 until 2023-10-14T01:11:55.463093+0000 (3600 sec) 2023-10-14T00:11:56.121 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/947637180 2023-10-14T00:11:56.135 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:11:56.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:56.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:11:56.677 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:56.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:11:56.685 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:56.688+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:11:56.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:56.732+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:11:56.733 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:56.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:11:56.733 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:56.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:11:56.733 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:11:56.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:11:57.308 INFO:tasks.workunit.client.0.smithi138.stderr:listed 14 entries 2023-10-14T00:11:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/947637180 2023-10-14T00:11:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:11:57.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:11:57.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:11:57.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:11:57.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:57.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:57.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:57.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:57.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:57.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:58.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:58.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:11:58.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:58.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:58.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:58.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:58.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:58.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:58.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:58.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:11:59.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:59.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:11:59.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:59.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:11:59.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:59.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:11:59.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:11:59.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.048+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.048+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8283780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.049 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.052+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.049 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.052+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8861700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.068 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.072+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.068 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.072+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8bfcc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.072+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.080+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8860980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.080+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.080+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.080+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.080+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deb100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.080+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.080+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dead00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.100+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.100+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.108+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.108+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deb700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.112+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.112+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dea380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.116+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.116+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deab00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.159 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.159 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae7880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.159 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.159 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deb980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8debb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8debd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.162 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.162 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.162 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.162 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.162 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8debf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.163 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.163 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.163 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.165 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.168+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.165 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:12:00.168+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:00.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:00.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:00.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:00.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:00.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:00.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:00.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:00.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:00.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:00.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:00.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:00.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:00.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:01.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:01.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:01.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:01.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:02.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:02.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:02.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:02.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:02.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:02.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:02.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:02.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:02.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:02.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:02.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:03.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:03.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:03.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:03.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:03.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:03.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:03.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:03.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:03.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:03.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:03.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:04.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:04.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:04.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:04.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:04.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:04.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:04.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:04.224+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:04.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:04.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:04.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:04.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:04.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:04.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:04.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:04.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:04.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:04.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:04.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:06.198 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:06.200+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:06.228 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:06.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:06.228 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:06.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:06.229 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:06.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:06.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:06.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:06.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:06.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:06.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:06.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:07.380 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:12:07.397 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:07.398 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:07.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:07.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:07.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:07.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:07.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:07.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:07.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:07.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:07.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:07.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:07.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:07.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:07.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:07.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:08.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:08.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:08.081 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:08.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:08.095 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:08.096+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:08.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:08.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:08.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:08.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:08.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:08.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:08.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:08.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:08.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:08.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:08.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:08.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:08.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:08.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:08.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:08.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:08.895 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/947637180 2023-10-14T00:12:08.896 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:12:08.896 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242328 2023-10-14T00:12:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/947637180 ]] 2023-10-14T00:12:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/947637180 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\4\7\6\3\7\1\8\0 ]] 2023-10-14T00:12:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:12:09.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.440+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:09.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:09.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:09.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:09.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:09.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:09.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:09.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:09.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:10.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:10.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:10.288 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:12:10.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:10.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:10.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:12:10.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:10.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:10.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:10.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:10.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:10.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:10.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:10.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:10.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:10.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:10.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:10.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:10.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.564+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:10.561 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.564+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:10.567 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:10.568 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:10.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:11.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:11.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:11.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:11.480+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:11.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:11.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:11.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:11.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:11.495 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:11.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:11.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:11.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:12.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:12.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:12.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:12.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:12.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:12.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:12.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:12.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:14.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:14.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:14.183 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:14.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:15.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:15.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:15.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:15.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:15.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:15.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:15.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:15.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:15.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:15.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:15.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:15.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:15.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:15.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:15.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:15.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:15.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:15.760+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:15.764 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:15.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:15.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:15.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:15.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:15.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:15.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:15.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:17.038 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:17.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:17.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:17.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:17.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:17.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:17.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:17.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:17.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:17.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:17.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:17.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:17.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:17.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:18.899 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:18.899 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:18.899 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:18.900 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:12:19.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:19.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:19.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:19.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:19.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:19.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:19.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:19.408+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:19.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:19.408+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:19.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:19.408+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:19.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:19.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:19.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:19.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:19.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:19.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:19.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:19.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:19.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:19.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:20.264 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/947637180 2023-10-14T00:12:20.264 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:12:20.269 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242340 2023-10-14T00:12:20.270 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:20.270 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/947637180 ]] 2023-10-14T00:12:20.270 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/947637180 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\4\7\6\3\7\1\8\0 ]] 2023-10-14T00:12:20.270 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:12:21.139 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.140+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:12:21.140 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.140+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:21.152 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:21.152 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:21.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:21.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:21.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:21.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:12:21.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:21.363 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:21.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:21.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:21.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:21.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:21.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:21.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:21.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:21.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:21.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:21.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:21.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:21.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:21.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:21.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:21.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:21.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:21.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:21.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:21.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:21.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:21.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:21.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:21.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:22.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:22.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:22.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:22.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:22.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:22.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:22.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:22.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:22.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:22.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:23.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:23.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:23.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:23.512+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:23.513 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:23.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:23.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:23.524+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:23.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:23.524+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:23.909 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:12:23.912+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:12:24.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:12:24.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:24.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:24.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:24.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:24.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:24.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:24.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:24.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:24.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:25.280 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:25.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:25.286 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:25.288+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:25.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:25.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:25.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:25.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:25.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:25.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:25.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:25.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:26.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:26.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:26.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:26.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:26.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:26.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:26.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:26.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:26.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:26.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:26.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:26.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:27.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:27.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:27.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:27.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:27.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:27.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:27.392 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:27.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:27.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:27.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:27.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:27.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:27.424+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:27.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:27.424+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:27.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:27.424+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:27.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:27.424+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:28.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:28.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:28.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:28.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:28.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:28.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:28.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:28.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:28.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:28.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:28.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:28.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:28.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:28.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:29.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:29.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:30.271 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:30.277 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:12:30.284 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:30.301 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:31.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:31.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/947637180 2023-10-14T00:12:31.405 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:12:31.406 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242351 2023-10-14T00:12:31.406 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:31.406 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/947637180 ]] 2023-10-14T00:12:31.406 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/947637180 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\4\7\6\3\7\1\8\0 ]] 2023-10-14T00:12:31.407 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:12:31.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.415+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:31.425 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:31.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:31.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:31.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:31.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:31.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:31.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:31.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:31.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:31.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:31.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:31.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:31.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:31.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:31.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:31.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:31.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:31.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:31.787 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:31.787 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:31.787 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:31.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:32.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:32.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:12:32.858 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:32.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:32.870 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:32.871+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:32.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:32.875+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:32.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:32.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:32.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:32.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:32.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:32.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:33.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:33.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:33.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:33.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:34.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:34.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:34.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:34.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:34.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:34.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:34.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:34.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:34.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:34.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:34.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:34.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:34.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:34.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:34.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:34.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:34.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:34.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:34.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:34.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:35.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:35.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:35.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:35.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:35.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:35.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:35.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:35.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:36.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:36.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:36.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:36.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:36.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:36.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:36.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:36.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:36.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:36.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:36.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:36.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:36.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:36.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:37.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:37.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:37.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:37.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:37.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:37.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:37.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:37.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:37.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:37.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:37.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:37.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:37.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:37.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:37.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:37.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:37.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:37.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:37.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:37.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:37.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:38.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:38.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:38.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:38.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:38.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:38.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:38.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:38.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:39.043 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:39.043+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:12:39.043 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:39.043+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:39.065 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:39.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:39.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:39.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:39.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:39.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:39.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:39.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:39.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:39.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:39.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:39.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:39.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:39.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:39.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:39.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:39.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:39.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:39.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:39.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:39.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:39.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:12:39.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:39.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:39.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:40.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:40.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:40.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:40.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:40.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:40.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:40.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:40.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:40.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:40.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:40.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:40.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:40.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:40.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:41.183 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:12:41.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:41.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:41.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:41.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:41.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:41.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:41.422 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:41.423 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:41.427 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:12:41.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.627+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:41.630 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:41.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:41.648 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:41.649 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:41.649 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:41.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:42.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:42.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:42.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:42.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:42.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:42.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:42.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:42.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:42.817 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/947637180 2023-10-14T00:12:42.817 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:12:42.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242362 2023-10-14T00:12:42.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:42.822 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/947637180 ]] 2023-10-14T00:12:42.822 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/947637180 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\4\7\6\3\7\1\8\0 ]] 2023-10-14T00:12:42.822 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:12:43.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:43.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:43.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:43.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:43.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:43.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:43.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:43.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:44.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:44.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:44.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:44.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:44.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:44.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:44.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:44.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:44.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:44.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:44.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:44.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:45.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:45.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:45.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:45.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:45.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:45.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:45.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:45.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:45.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:45.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:45.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:45.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:45.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:46.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:46.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:46.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:46.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:46.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:46.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:46.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:46.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:49.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:49.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:49.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:49.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:49.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:49.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:49.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:49.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:49.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:49.343+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:12:49.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:49.343+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:49.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:49.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:49.392 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:49.395+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:49.411 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:49.411+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:49.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:49.411+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:49.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:49.411+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:49.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:49.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:49.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:49.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:49.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:49.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:49.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:49.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:49.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:49.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:50.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:50.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:50.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:50.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:50.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:50.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:50.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:50.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:50.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:50.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:50.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:50.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:50.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:50.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:50.885 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:50.887+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:12:50.885 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:50.887+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:12:50.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:50.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:50.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:50.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:12:50.931 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:50.931+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:50.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:50.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:50.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:50.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:51.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:51.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:51.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:51.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:51.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:51.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:51.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:51.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:51.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:51.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:52.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:52.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:52.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:52.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:52.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:52.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:52.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:52.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:52.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:52.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:52.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:52.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:52.828 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:12:52.828 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:52.831 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:52.869 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:52.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:52.927+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:52.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:52.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:52.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:52.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:52.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:52.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:53.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:53.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:12:53.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:53.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:12:53.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:53.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:12:53.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:12:53.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:12:53.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:53.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:54.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:54.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:54.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:54.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:54.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:54.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:54.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:54.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:54.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:54.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:54.435 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/947637180 2023-10-14T00:12:54.454 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:12:54.456 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242374 2023-10-14T00:12:54.456 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:54.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/947637180 ]] 2023-10-14T00:12:54.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/947637180 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\4\7\6\3\7\1\8\0 ]] 2023-10-14T00:12:54.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:12:57.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:12:57.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:57.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:12:57.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:12:57.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:57.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:57.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:57.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:57.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:58.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:58.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:58.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:58.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:58.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:58.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:12:59.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:59.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:12:59.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:59.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:12:59.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:59.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:12:59.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:59.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:12:59.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:12:59.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:00.349 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:00.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:00.350 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:00.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:00.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:00.371+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:00.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:00.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:00.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:00.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:00.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:00.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:00.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:00.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:02.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:02.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:02.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:02.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:02.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:02.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:02.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:02.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:02.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:02.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:02.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:02.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:04.461 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:04.463 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:04.464 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:04.468 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:13:05.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:05.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:05.296 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1188183334 2023-10-14T00:13:05.296 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:13:05.297 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242385 2023-10-14T00:13:05.297 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:05.297 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1188183334 ]] 2023-10-14T00:13:05.297 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1188183334 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\4\7\6\3\7\1\8\0 ]] 2023-10-14T00:13:05.298 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/1188183334 2023-10-14T00:13:07.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:07.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:07.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:07.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:07.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:07.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:07.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:07.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:07.068 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/1188183334 until 2023-10-14T01:13:06.281930+0000 (3600 sec) 2023-10-14T00:13:07.110 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:13:07.110 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/1188183334 2023-10-14T00:13:07.796 INFO:tasks.workunit.client.0.smithi138.stderr:listed 15 entries 2023-10-14T00:13:07.831 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/1188183334 2023-10-14T00:13:07.832 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:13:11.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:11.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:11.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:11.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:11.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:11.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:11.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:11.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:13.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:13.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:13.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:13.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:13.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:13.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:13.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:13.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:13.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:13.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:13.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:13.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:13.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:13.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:14.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:14.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:14.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:14.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:14.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:14.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:15.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:15.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:15.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:15.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:15.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:15.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:15.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:15.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:15.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:15.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:15.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:15.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:16.725 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:16.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:16.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:16.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:16.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:16.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:16.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:16.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:16.754 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:16.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:16.754 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:16.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:17.833 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:13:17.839 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:17.841 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:17.852 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:18.763 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1188183334 2023-10-14T00:13:18.763 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:13:18.764 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242398 2023-10-14T00:13:18.764 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:18.764 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1188183334 ]] 2023-10-14T00:13:18.765 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1188183334 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\8\8\1\8\3\3\3\4 ]] 2023-10-14T00:13:18.765 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:13:20.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:20.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:20.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:20.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:20.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:20.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:21.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:21.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:21.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:21.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:21.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:21.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:21.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:21.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:21.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:21.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:21.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:21.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:21.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:21.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:21.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:21.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:22.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:22.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:22.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:22.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:22.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:22.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:22.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:22.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:22.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:22.819 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:22.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:22.820 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:22.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:22.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:22.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:22.840 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:22.838+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:22.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:22.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:22.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:22.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:22.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:22.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:23.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:23.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:23.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:23.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:23.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:23.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:24.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:24.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:24.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:24.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:24.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:24.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:24.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:24.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:25.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:25.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:25.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:25.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:25.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:25.303 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:25.311 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:25.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.318+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:25.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:25.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:25.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:25.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:13:25.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:13:25.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:25.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:25.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:25.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:25.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:25.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:25.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:25.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:25.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:25.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:25.885 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.886+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:25.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:25.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:25.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:25.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:26.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:26.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:26.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:26.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:26.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:26.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:26.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:26.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:26.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:27.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:27.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:27.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:27.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:27.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:27.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:27.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:27.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:27.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:27.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:27.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:27.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:27.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:27.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:27.754 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:27.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:27.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:27.770+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:27.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:27.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:27.795 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:27.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:27.795 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:27.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:27.795 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:27.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:28.774 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:13:28.775 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:28.777 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:28.785 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1188183334 2023-10-14T00:13:29.992 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:13:29.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242409 2023-10-14T00:13:29.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1188183334 ]] 2023-10-14T00:13:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1188183334 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\8\8\1\8\3\3\3\4 ]] 2023-10-14T00:13:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:13:31.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:31.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:31.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:31.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:31.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:31.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:31.254 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:31.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:31.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:31.646+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:31.672 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:31.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:31.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:31.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:32.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:32.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:32.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:32.385 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:32.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:32.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:32.521 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:32.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:32.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:32.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:32.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:32.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:32.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:32.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:32.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:32.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:32.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:32.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:32.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:32.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:32.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:32.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:32.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:32.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:32.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:32.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:33.600 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:33.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:33.619 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:33.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:33.624 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:33.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:33.624 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:33.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:33.624 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:33.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:33.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:33.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:34.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:34.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:34.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:34.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:34.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:34.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:34.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:34.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:34.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:34.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:34.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:34.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:34.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:34.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:35.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:35.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:13:35.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:35.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:35.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:35.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:35.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:35.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:35.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:35.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:35.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:35.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:36.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:36.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:36.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:36.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:36.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:36.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:36.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:36.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:36.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:36.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:36.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:36.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:36.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:36.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:36.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:36.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:36.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:36.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:38.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:38.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:38.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:38.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:38.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:38.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:38.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:38.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:38.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:38.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:39.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:39.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:39.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:39.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:39.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:39.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:39.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:39.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:39.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:39.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:39.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:39.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:39.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:39.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:39.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:39.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:39.593 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:39.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:39.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:39.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:39.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:39.622+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:39.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:39.622+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:39.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:39.622+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:40.004 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:13:40.004 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:40.014 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:40.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:40.088 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:40.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:40.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:40.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:40.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:40.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:13:40.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:40.535 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:40.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:40.573 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:40.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:40.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:40.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:40.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:40.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:40.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:40.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:40.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:40.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:41.422 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1188183334 2023-10-14T00:13:41.423 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:13:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242421 2023-10-14T00:13:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1188183334 ]] 2023-10-14T00:13:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1188183334 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\8\8\1\8\3\3\3\4 ]] 2023-10-14T00:13:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:13:41.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:41.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:41.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:41.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:41.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:41.788 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.790+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:13:41.798 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:41.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:41.831 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.830+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:41.831 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.830+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:41.831 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.830+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:41.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:41.830+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:42.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:42.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:42.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:42.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:42.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:42.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:42.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:42.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:42.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:42.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:42.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:42.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:44.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:44.087 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:44.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:44.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:44.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:44.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:44.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:44.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:44.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:45.445 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:13:45.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:45.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:45.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:45.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:45.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:45.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:45.667 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:45.667 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:45.667 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:45.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:48.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:48.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:48.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:48.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:48.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:48.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:48.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:48.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:49.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:49.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:49.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:49.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:49.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:49.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:49.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:49.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:49.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:49.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:49.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:50.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:50.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:50.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:50.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:50.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:50.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:50.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:50.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:50.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:50.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:50.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:51.426 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:13:51.427 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:51.429 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:51.465 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:52.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:52.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:52.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:52.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:52.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:52.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:52.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:52.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:52.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:52.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:53.013 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1188183334 2023-10-14T00:13:53.025 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:13:53.026 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242433 2023-10-14T00:13:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1188183334 ]] 2023-10-14T00:13:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1188183334 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\8\8\1\8\3\3\3\4 ]] 2023-10-14T00:13:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:13:54.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:54.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:13:54.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:54.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:13:54.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:54.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:13:54.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:54.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:13:54.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:54.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:13:54.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:13:54.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:13:54.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:54.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:54.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:54.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:54.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:54.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:54.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:54.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:55.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:55.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:55.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:55.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:55.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:55.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:58.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:58.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:58.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:58.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:58.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:58.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:13:58.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:58.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:58.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:58.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:58.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:58.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:58.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:58.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:59.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:59.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:13:59.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:59.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:59.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:13:59.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:13:59.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:13:59.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:13:59.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:13:59.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:13:59.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:13:59.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:00.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:00.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:00.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:00.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:00.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:00.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:03.031 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:03.031 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:14:03.032 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:03.033 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:03.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:03.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:03.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:03.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:03.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:03.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:03.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:03.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:03.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:03.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:03.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:03.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:03.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:03.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:04.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:04.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:04.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:04.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:04.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:04.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1188183334 2023-10-14T00:14:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:14:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242444 2023-10-14T00:14:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1188183334 ]] 2023-10-14T00:14:04.278 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1188183334 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\8\8\1\8\3\3\3\4 ]] 2023-10-14T00:14:04.278 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:14:05.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:05.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:14:05.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:05.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:05.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:05.129+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:05.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:05.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:05.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:05.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:05.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:05.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:05.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:05.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:06.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:06.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:06.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:06.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:06.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:06.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:06.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:06.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:06.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:06.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:06.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:06.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:06.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:06.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:06.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:06.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:06.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:06.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:14:06.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:06.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:06.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:06.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:06.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:06.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:06.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:06.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:07.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:07.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:07.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:07.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:07.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:07.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:07.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:07.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:07.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:07.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:07.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:07.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:07.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:07.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:08.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:08.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:08.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:08.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:08.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:08.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:08.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:08.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:08.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:08.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:08.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:08.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:09.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.037+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:09.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.053+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:09.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:09.081 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:09.081 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:09.081 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:09.858 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.857+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:14:09.859 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.857+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:09.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.865+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:09.870 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.869+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:09.875 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:09.875 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:09.875 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:09.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:12.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:12.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:12.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:12.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:12.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:12.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:13.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:13.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:13.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:13.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:13.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:13.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:13.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:13.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:13.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:13.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:13.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:14.289 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:14.292 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:14.295 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:14.303 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:14:14.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:14.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:14.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:14.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:14.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:14.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:14.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:14.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:14.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:14.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:14.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:14.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:15.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:15.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:15.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:15.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:15.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:15.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:15.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:15.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:15.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:15.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:15.492 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2052584210 2023-10-14T00:14:15.494 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:14:15.495 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242455 2023-10-14T00:14:15.496 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:15.496 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2052584210 ]] 2023-10-14T00:14:15.496 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2052584210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\8\8\1\8\3\3\3\4 ]] 2023-10-14T00:14:15.496 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2052584210 2023-10-14T00:14:16.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:14:16.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:16.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:16.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.461+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:16.485 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:16.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:16.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:16.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:14:16.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:14:16.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:16.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:16.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:16.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:16.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:17.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:17.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:17.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:17.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:17.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:14:17.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:17.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:17.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:17.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:17.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:17.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:17.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:18.047 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2052584210 until 2023-10-14T01:14:17.442056+0000 (3600 sec) 2023-10-14T00:14:18.135 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:14:18.135 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2052584210 2023-10-14T00:14:19.318 INFO:tasks.workunit.client.0.smithi138.stderr:listed 16 entries 2023-10-14T00:14:19.367 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2052584210 2023-10-14T00:14:19.367 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:14:20.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:20.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:24.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:24.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:24.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:24.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:24.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:24.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:24.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:24.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:24.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:24.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:24.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:24.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:24.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:24.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:25.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:25.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:25.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:25.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:25.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:25.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:25.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:25.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:25.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:25.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:27.936 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:27.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:14:27.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:27.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:27.949 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:27.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:27.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:27.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:27.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:27.981+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:27.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:27.981+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:27.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:27.981+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:29.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:29.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:29.368 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:14:29.369 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:29.374 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:29.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:29.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:29.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:29.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:29.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:29.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:29.415 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:29.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:29.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:29.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:29.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:30.682 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2052584210 2023-10-14T00:14:30.683 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:14:30.684 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242470 2023-10-14T00:14:30.684 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:30.685 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2052584210 ]] 2023-10-14T00:14:30.685 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2052584210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\0\5\2\5\8\4\2\1\0 ]] 2023-10-14T00:14:30.685 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:14:31.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:31.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:31.627 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:31.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:33.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:33.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:33.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:33.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:33.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:33.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:33.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:33.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:33.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:33.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:33.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:33.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:35.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:35.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:35.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:35.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:35.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:35.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:35.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:35.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:35.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:35.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:35.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:14:35.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:35.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:35.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:35.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:35.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:35.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:35.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:35.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:35.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:35.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:35.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:35.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:36.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:36.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:36.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:36.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:36.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:36.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:36.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:36.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:36.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:36.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:36.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:36.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:36.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:36.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:37.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:37.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:37.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:37.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:37.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:37.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:37.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:37.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:37.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:37.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:37.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:37.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:38.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:38.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:38.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:38.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:38.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:38.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:38.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:38.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:38.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:38.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:38.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:38.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:40.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:40.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:40.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:40.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:40.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:40.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:14:40.700 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:40.702 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:40.703 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:41.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:41.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:41.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:41.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:41.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:41.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:41.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:41.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:41.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:41.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:42.332 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2052584210 2023-10-14T00:14:42.332 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:14:42.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242482 2023-10-14T00:14:42.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:42.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2052584210 ]] 2023-10-14T00:14:42.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2052584210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\0\5\2\5\8\4\2\1\0 ]] 2023-10-14T00:14:42.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:14:42.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:42.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:42.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:42.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:42.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:42.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:42.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:42.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:42.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:42.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:43.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:43.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:43.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:43.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:43.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:43.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:43.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:43.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:43.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:44.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:44.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:44.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:44.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:44.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:44.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:44.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:44.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:44.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:45.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:45.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:45.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:45.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:45.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:45.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:45.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:45.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:45.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:48.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:48.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:48.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:48.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:48.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:48.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:48.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:48.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:48.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:48.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:48.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:48.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:48.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:48.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:48.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:48.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:48.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:48.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:48.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:48.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:48.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:48.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:48.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:48.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:49.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:49.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:49.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:49.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:49.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:49.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:49.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:49.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:49.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:49.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:49.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:49.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:50.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:50.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:50.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:50.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:50.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:50.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:51.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:51.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:51.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:51.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:51.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:51.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:51.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:51.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:51.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:51.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:52.335 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:14:52.335 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:52.346 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:52.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:52.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:52.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:52.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:52.367 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:52.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:52.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:53.961 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2052584210 2023-10-14T00:14:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:14:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242493 2023-10-14T00:14:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:53.964 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2052584210 ]] 2023-10-14T00:14:53.964 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2052584210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\0\5\2\5\8\4\2\1\0 ]] 2023-10-14T00:14:53.964 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:14:54.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:54.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:54.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:54.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:54.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:54.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:54.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:54.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:54.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:54.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:55.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:55.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:55.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:55.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:55.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:55.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:14:55.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:55.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:56.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:14:56.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:56.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:56.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:56.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:56.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:56.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:56.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:56.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:56.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:56.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:14:56.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:56.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:56.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:56.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:56.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:56.868+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:56.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:56.868+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:57.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:57.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:57.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:57.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:57.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:57.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:57.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:57.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:57.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:57.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:57.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:57.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:58.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:58.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:14:58.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:58.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:14:58.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:58.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:14:58.212 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:58.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:14:58.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:58.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:58.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:58.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:58.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:58.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:58.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:58.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:14:58.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:58.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:14:58.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:14:58.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:14:58.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:58.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:58.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:58.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:58.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:58.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:59.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:14:59.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:14:59.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:14:59.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:14:59.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:59.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:59.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:14:59.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:14:59.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:14:59.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:14:59.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:00.006 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.008+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.008+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.008+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8464e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.008+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deb600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.012+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.012+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8861180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.043 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.044+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.044+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.044+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8861f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.044+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.064+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.064+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd7f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.072+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.072+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8861c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.088+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.088+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.088+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfd580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.088 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.088+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd7300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.096+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.096+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b60e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.100+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.100+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8282900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.122 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.124+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.122 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.124+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aaf080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.127 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.124+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.127 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.124+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.151 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.152+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.151 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.152+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.151 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.152+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905d400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.151 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.152+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8878000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.156 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.156+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.156+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.157 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.169 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.172+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.170 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.172+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.170 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.172+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.170 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.172+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d905d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.195 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.196+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.195 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:15:00.196+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9108b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:15:00.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:00.836+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:00.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:00.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:00.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:00.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:00.858 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:00.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:00.858 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:00.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:00.859 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:00.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:01.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:01.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:15:01.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:01.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:01.038 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:01.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:01.048 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:01.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:01.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:01.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:01.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:01.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:03.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:03.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:03.963 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:15:03.964 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:03.970 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:03.980 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:04.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:04.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:04.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:04.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:04.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:04.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:04.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:04.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:04.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:04.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:04.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:04.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:04.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:04.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:05.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:15:05.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:05.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:05.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:05.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:05.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:05.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:05.434 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2052584210 2023-10-14T00:15:05.435 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:15:05.436 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242505 2023-10-14T00:15:05.436 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:05.437 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2052584210 ]] 2023-10-14T00:15:05.437 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2052584210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\0\5\2\5\8\4\2\1\0 ]] 2023-10-14T00:15:05.437 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:15:05.864 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:05.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:05.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:05.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:05.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:05.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:05.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:06.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:06.512+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:15:06.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:06.512+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:06.517 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:06.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:06.567 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:06.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:06.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:06.576+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:06.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:06.576+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:06.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:06.576+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:08.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:08.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:08.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:08.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:08.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:08.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:08.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:08.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:08.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:08.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:08.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:08.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:08.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:08.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:10.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:10.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:10.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:10.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:10.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:10.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:10.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:10.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:10.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:10.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:10.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:10.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:10.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:10.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:10.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:15:10.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:10.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:10.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:10.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:10.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:10.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:11.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:11.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:11.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:11.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:11.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:11.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:11.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:11.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:11.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:11.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:11.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:11.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:12.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:12.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:12.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:12.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:12.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:12.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:12.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:12.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:12.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:12.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:12.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:12.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:12.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:12.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:14.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:14.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:14.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:14.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:14.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:14.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:14.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:14.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:14.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:15.445 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:15.446 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:15:15.447 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:15.478 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:15.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:15.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:15.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:15.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:15.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:15.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:15.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:15.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:15.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:15.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:16.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:16.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:16.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:16.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:16.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:16.592 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2052584210 2023-10-14T00:15:16.593 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:15:16.593 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242516 2023-10-14T00:15:16.594 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:16.594 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2052584210 ]] 2023-10-14T00:15:16.594 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2052584210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\0\5\2\5\8\4\2\1\0 ]] 2023-10-14T00:15:16.594 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:15:16.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:16.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:16.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:16.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:16.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:16.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:17.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:17.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:17.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:17.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:17.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:17.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:17.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:17.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:17.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:17.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:17.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:17.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:20.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:20.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:20.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:20.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:20.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:20.927+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:20.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:20.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:20.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:20.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:20.971 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:20.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:20.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:20.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:20.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:20.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:21.358 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:21.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:15:21.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:21.363+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:21.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:21.371+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:21.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:21.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:21.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:21.399+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:21.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:21.399+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:21.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:21.399+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:24.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:24.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:24.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:24.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:24.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:24.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:24.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:24.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:24.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:24.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:24.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:24.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:24.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:24.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:24.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:24.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:24.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:24.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:24.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:24.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:24.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:25.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:25.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:25.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:25.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:25.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:25.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:25.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:25.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:25.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:25.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:26.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:26.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:26.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:26.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:26.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:26.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:26.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:26.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:26.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:26.160 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:26.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.319+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:26.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.319+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:26.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:26.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:26.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:26.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:26.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:26.595 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:15:26.595 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:26.613 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:26.615 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:26.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:26.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:26.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:26.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:26.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:26.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:27.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:27.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:27.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:27.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:27.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:27.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:27.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:27.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:27.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:27.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:27.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:28.048 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3062435210 2023-10-14T00:15:28.061 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:15:28.068 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242528 2023-10-14T00:15:28.069 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:28.069 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3062435210 ]] 2023-10-14T00:15:28.069 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3062435210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\0\5\2\5\8\4\2\1\0 ]] 2023-10-14T00:15:28.069 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3062435210 2023-10-14T00:15:28.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:28.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:28.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:28.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:28.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:28.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:28.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:28.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:28.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:28.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:28.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:28.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:30.097 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3062435210 until 2023-10-14T01:15:29.508527+0000 (3600 sec) 2023-10-14T00:15:30.154 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:15:30.157 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3062435210 2023-10-14T00:15:30.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:30.279+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:30.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:30.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:30.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:30.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:30.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:30.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:30.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:30.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:30.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:30.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:31.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:31.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:31.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:31.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:31.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:31.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:31.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:31.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:31.378 INFO:tasks.workunit.client.0.smithi138.stderr:listed 17 entries 2023-10-14T00:15:31.436 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3062435210 2023-10-14T00:15:31.436 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:15:33.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:33.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:33.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:33.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:33.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:33.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:33.735+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:33.763 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:33.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:33.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:33.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:33.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:33.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:33.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:33.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:33.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:33.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:34.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:34.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:34.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:34.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:34.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:34.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:34.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:34.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:34.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:37.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:37.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:37.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:37.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:38.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:38.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:39.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:39.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:39.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:39.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:39.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:39.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:39.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:39.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:39.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:39.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:39.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:39.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:40.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:40.083+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:40.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:40.095+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:40.121 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:40.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:40.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:40.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:40.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:40.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:40.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:40.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:40.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:40.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:40.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:40.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:40.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:40.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:41.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:40.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:15:41.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:41.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:41.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:41.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:41.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:15:41.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:41.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:41.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:41.031+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:41.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:41.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:41.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:41.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:41.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:41.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:41.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:41.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:41.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:41.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:41.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:41.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:41.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:41.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:41.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:41.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:41.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:41.437 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:15:41.438 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:41.459 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:41.459 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:42.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:42.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:42.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:42.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:42.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:42.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:42.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:42.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:42.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:42.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:42.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:42.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:42.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:42.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:42.623 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3062435210 2023-10-14T00:15:42.633 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:15:42.635 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242542 2023-10-14T00:15:42.635 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:42.635 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3062435210 ]] 2023-10-14T00:15:42.636 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3062435210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\0\6\2\4\3\5\2\1\0 ]] 2023-10-14T00:15:42.636 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:15:43.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:43.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:43.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:43.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:43.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:43.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:43.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:43.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:43.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:43.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:43.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:43.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:44.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:44.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:44.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:44.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:44.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:44.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:44.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:44.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:44.513 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:44.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:44.513 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:44.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:44.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:44.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:44.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:44.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:44.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:44.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:44.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:44.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:44.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:44.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:44.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:44.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:45.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:15:45.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:45.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:45.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:45.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:45.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:45.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:45.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:45.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:47.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:47.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:47.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:47.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:47.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:47.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:47.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:47.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:47.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:47.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:47.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:47.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:47.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:47.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:47.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:49.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:49.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:50.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:50.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:50.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:50.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:50.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:50.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:15:50.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:50.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:50.463 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:50.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:50.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:50.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:50.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:50.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:50.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:50.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:50.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:50.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:50.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:50.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:50.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:50.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:51.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:51.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:51.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:51.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:52.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:52.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:52.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:52.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:52.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:52.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:52.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:52.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:52.637 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:52.637 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:15:52.642 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:52.665 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:53.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:53.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:53.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:53.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:53.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:53.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:53.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:53.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:53.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:53.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:53.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:53.885 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3062435210 2023-10-14T00:15:53.895 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:15:53.913 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242553 2023-10-14T00:15:53.913 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:53.914 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3062435210 ]] 2023-10-14T00:15:53.914 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3062435210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\0\6\2\4\3\5\2\1\0 ]] 2023-10-14T00:15:53.914 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:15:54.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:54.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:54.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:54.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:54.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:54.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:54.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:54.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:54.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:54.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:54.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:54.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:54.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:55.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:55.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:55.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:55.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:55.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:55.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:55.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:55.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:55.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:55.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:55.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:55.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:15:55.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:55.830+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:15:55.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:55.830+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:55.843 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:55.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:57.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:57.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:57.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:57.418+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:57.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:57.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:57.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:57.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:57.437 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:57.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:15:57.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:57.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:15:57.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:57.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:15:57.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:57.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:15:57.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:57.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:15:57.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:57.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:15:57.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:57.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:15:57.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:15:57.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:15:58.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:58.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:15:59.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:59.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:15:59.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:59.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:15:59.030 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:59.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:15:59.030 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:59.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:15:59.030 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:59.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:15:59.030 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:59.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:15:59.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:15:59.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:00.002 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.002+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.002 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.002+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8aaf080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.005 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.006+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.006 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.006+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8282600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8283680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905d180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.042+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.049 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.049 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deb000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.056 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.054+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.056 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.054+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.059 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.058+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.059 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.058+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.062+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.062+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.074+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.074+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.074+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.074+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.074+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:16:00.074+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:16:00.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:00.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:00.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:00.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:00.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:00.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:00.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:00.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:00.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:00.454 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:00.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:00.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:00.458+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:00.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:00.458+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:01.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:01.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:16:01.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:01.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:16:02.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:02.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:02.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:02.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:02.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:02.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:02.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:02.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:02.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:02.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:02.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:03.925 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:03.926 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:03.927 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:03.927 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:16:05.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:16:05.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:05.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:05.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:05.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:05.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:05.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:05.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:05.395 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3062435210 2023-10-14T00:16:05.405 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:16:05.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:05.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:05.413 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242565 2023-10-14T00:16:05.413 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:05.413 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3062435210 ]] 2023-10-14T00:16:05.413 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3062435210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\0\6\2\4\3\5\2\1\0 ]] 2023-10-14T00:16:05.414 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:16:05.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:05.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:05.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:05.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:05.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:16:05.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:05.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:05.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:05.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:05.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:05.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:05.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:05.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:05.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:05.593 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:05.590+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:05.600 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:05.598+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:05.619 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:05.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:05.619 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:05.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:05.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:05.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:06.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:06.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:06.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:06.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:06.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:06.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:06.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:06.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:06.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:06.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:06.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:06.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:07.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:07.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:07.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:07.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:07.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:07.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:07.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:07.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:07.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:07.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:08.096 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:08.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:08.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:08.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:08.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:08.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:09.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:09.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:10.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:10.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:10.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:10.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:10.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:10.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:10.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:10.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:10.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:10.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:10.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:10.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:10.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:10.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:10.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:10.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:10.834 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:10.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:10.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:10.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:10.875 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:10.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:10.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:10.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:10.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:10.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:11.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:11.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:11.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:11.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:11.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:11.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:11.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:11.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:11.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:11.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:11.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:11.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:11.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:11.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:11.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:11.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:11.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:12.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:12.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:12.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:12.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:12.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:12.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:12.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:12.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:12.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:12.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:13.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:13.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:13.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:13.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:13.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:13.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:13.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:13.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:13.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:14.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.638+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:14.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:14.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:14.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:14.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:14.689 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:14.689 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:14.706 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:14.726 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:14.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:15.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:15.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:15.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.294+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:15.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:15.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:15.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:15.429 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:15.430 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:16:15.463 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:15.656 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:15.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:15.670+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:16.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:16.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:16.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:16.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:16.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:16.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:16.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:16.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:16.778 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3062435210 2023-10-14T00:16:16.779 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:16:16.780 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242576 2023-10-14T00:16:16.780 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3062435210 ]] 2023-10-14T00:16:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3062435210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\0\6\2\4\3\5\2\1\0 ]] 2023-10-14T00:16:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:16:18.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:18.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:18.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:18.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:18.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:18.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:18.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:18.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:18.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:18.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:18.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:18.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:18.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:19.141 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.138+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:19.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.138+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:19.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:19.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:19.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:19.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:19.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:19.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:19.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:19.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:19.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:19.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:19.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:19.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.922+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:19.959 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:19.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:20.250 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:20.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:20.250 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:20.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:20.259 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:20.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:20.279 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:20.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:20.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:20.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:20.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:20.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:20.286 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:20.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:21.110 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:21.126 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:21.143 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:21.150 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:21.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:21.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:21.628 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:21.629 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.630+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:21.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:21.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:21.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:21.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:21.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:21.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:21.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:21.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:21.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:21.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:21.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:21.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:21.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:21.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:21.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:21.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:25.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:25.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:26.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:26.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:26.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:26.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:26.311 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:26.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:26.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:26.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:26.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:26.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:26.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:26.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:26.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:26.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:26.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:26.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:26.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:26.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:26.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:26.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:26.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:26.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:26.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:26.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:26.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:26.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:26.782 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:16:26.783 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:26.783 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:26.809 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:28.023 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3062435210 2023-10-14T00:16:28.023 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:16:28.024 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242588 2023-10-14T00:16:28.024 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:28.024 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3062435210 ]] 2023-10-14T00:16:28.024 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3062435210 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\0\6\2\4\3\5\2\1\0 ]] 2023-10-14T00:16:28.025 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:16:28.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:28.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:28.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:28.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:28.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:28.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:28.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:28.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:28.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:28.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:29.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:29.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:29.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:29.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:29.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:29.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:29.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:29.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:29.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:29.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:29.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:29.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:29.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:30.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:30.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:30.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:30.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:30.525 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:30.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:30.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:30.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:30.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:30.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:30.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:30.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:30.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:30.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:30.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:30.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:30.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:30.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:30.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:30.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:30.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:30.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:30.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:30.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:30.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:31.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:30.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:16:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:16:33.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:16:33.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:33.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:33.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:33.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:33.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:33.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:33.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:33.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:33.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:33.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:33.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:33.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:33.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:34.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:34.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:33.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:34.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:34.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:34.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:34.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:34.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:34.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:34.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:34.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:34.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:34.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:35.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:35.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:35.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:35.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:35.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:35.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:35.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:35.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:35.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:35.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:35.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:35.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:37.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:37.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:37.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:37.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:37.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:37.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:37.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:37.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:37.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:37.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:37.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:37.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:37.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:37.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:38.030 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:38.030 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:16:38.037 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:38.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:38.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:38.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:38.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:38.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:38.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:38.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:38.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:38.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:38.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:38.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:38.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:38.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:39.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:16:39.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:16:39.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:16:39.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:39.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:39.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:39.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:39.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:39.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:39.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:39.541 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/967096462 2023-10-14T00:16:39.545 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:16:39.546 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242599 2023-10-14T00:16:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/967096462 ]] 2023-10-14T00:16:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/967096462 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\0\6\2\4\3\5\2\1\0 ]] 2023-10-14T00:16:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/967096462 2023-10-14T00:16:40.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:40.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:40.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:40.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:40.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:40.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:40.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:40.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:40.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:40.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:40.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:40.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:40.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:40.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:42.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.093+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:42.095 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.093+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:42.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:42.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.133+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:42.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:42.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:42.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:42.163 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/967096462 until 2023-10-14T01:16:41.556411+0000 (3600 sec) 2023-10-14T00:16:42.229 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/967096462 2023-10-14T00:16:42.230 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:16:42.434 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.433+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:42.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.433+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:42.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:42.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.461+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:42.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:42.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:42.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:42.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:42.686 INFO:tasks.workunit.client.0.smithi138.stderr:2023-10-14T00:16:42.685+0000 7f3cf91a6640 0 --2- 172.21.15.138:0/2772527458 >> v2:172.21.15.138:3300/0 conn(0x7f3cf406c320 0x7f3cf406c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-14T00:16:43.663 INFO:tasks.workunit.client.0.smithi138.stderr:listed 18 entries 2023-10-14T00:16:43.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:43.689+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:43.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/967096462 2023-10-14T00:16:43.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:16:43.719 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:43.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:43.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:43.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:43.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:43.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:44.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:44.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:44.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:44.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:44.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:44.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:44.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:44.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:44.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:44.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:44.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:44.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:44.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:44.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:45.629 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:45.657 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:45.673 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:45.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:45.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:45.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:45.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:45.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:45.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:45.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:45.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:45.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:45.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:46.419 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:46.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:46.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:46.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:46.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:46.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:46.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:46.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:46.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:46.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:47.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:47.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:47.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:47.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:47.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:47.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:47.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:47.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:47.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:47.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:47.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:47.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:49.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:49.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:49.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:49.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:49.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:49.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:49.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:49.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:49.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:49.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:49.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:50.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:50.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:50.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:50.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:50.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:50.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:50.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:50.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:53.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:53.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:53.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:53.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:53.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:53.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:53.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:53.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:53.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:53.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:53.720 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:53.721 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:53.724 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:53.751 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:16:55.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:55.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:55.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:55.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:55.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:55.194 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/967096462 2023-10-14T00:16:55.200 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:16:55.201 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242615 2023-10-14T00:16:55.201 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:55.201 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/967096462 ]] 2023-10-14T00:16:55.201 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/967096462 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\6\7\0\9\6\4\6\2 ]] 2023-10-14T00:16:55.202 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:16:55.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:55.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:55.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:55.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:56.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:56.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:56.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:56.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:56.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:56.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:56.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:56.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:56.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:56.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:56.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:16:56.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:56.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:56.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:56.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:56.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:56.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:56.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:56.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:57.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:57.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:16:57.850 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:57.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:16:57.864 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:57.861+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:16:57.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:57.865+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:16:57.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:57.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:16:57.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:57.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:16:57.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:16:57.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:16:58.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:16:58.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:16:58.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:58.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:58.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:58.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:58.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:59.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:16:59.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:59.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:59.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:16:59.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:16:59.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:16:59.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:16:59.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:16:59.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:16:59.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:16:59.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:00.002 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.001+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.002 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.001+0000 7f082630d640 -1 librbd::ImageState: 0x5627d81a9b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.005+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.005+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8861780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.013+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.013+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b18d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.013+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.013+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8861400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.029+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.029+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.033+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.033+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.034 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.037+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.034 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.037+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.036 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.037+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.037 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.037+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905df00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.053+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.053+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.057+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.058 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.057+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.061+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.061+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae7680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.077+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.077+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.077+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dea780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.077+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.093+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.093+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.093+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.093+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.101 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.101+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.101 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.101+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.101 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.101+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.101 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.101+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.101+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.101+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905fc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.109+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.109+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.109+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:00.109+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:00.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:00.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:00.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:00.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:00.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:00.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:00.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:00.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:00.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:00.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:00.509+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:00.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:00.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:00.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:00.521+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:00.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:00.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:00.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:00.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:00.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:00.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:00.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:00.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:01.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:01.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:01.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:01.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:01.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:01.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:02.030 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:02.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:17:02.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:02.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:02.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:02.041+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:02.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:02.053+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:02.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:02.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:02.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:02.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:02.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:02.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:02.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:02.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:02.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:02.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:02.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:03.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:03.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:17:03.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:03.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:03.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:03.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:03.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:03.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:03.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:03.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:03.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:03.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:03.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:03.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:04.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:04.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:04.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:04.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:04.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:04.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:05.212 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:05.212 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:17:05.217 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:05.218 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:06.886 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/967096462 2023-10-14T00:17:06.891 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:17:06.892 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242626 2023-10-14T00:17:06.892 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:06.893 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/967096462 ]] 2023-10-14T00:17:06.893 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/967096462 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\6\7\0\9\6\4\6\2 ]] 2023-10-14T00:17:06.893 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:17:08.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:08.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:08.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:08.817+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:08.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:08.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:08.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:08.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:09.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:09.769+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:17:09.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:09.769+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:09.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:09.809+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:09.812 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:09.813+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:09.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:09.945+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:09.986 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:09.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:11.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:11.242 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:11.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.249+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:11.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:11.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:11.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:11.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:11.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.857+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:11.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:11.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:11.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:12.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:17:12.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:12.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:12.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:12.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:12.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:12.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:12.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:12.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:12.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:12.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:12.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:12.597 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:12.597+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:12.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:12.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:12.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:12.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:12.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:12.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:13.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:13.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:13.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:13.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:13.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:13.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:13.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:13.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:13.617 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:13.621+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:13.618 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:13.621+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:13.618 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:13.621+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:13.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:13.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:13.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:13.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:13.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:13.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:13.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:13.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:13.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:14.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:14.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:14.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:14.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:14.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:14.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:14.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:14.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:14.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:14.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:14.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:14.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:15.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:15.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:15.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:15.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:15.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:15.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:15.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:15.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:15.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:15.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:15.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:15.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:15.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:15.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:16.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:16.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:16.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:16.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:16.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:16.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:16.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:16.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:16.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:16.372+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:17:16.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:16.372+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:16.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:16.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:16.906 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:16.907 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:17:16.907 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:16.931 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:18.364 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/967096462 2023-10-14T00:17:18.371 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:17:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242638 2023-10-14T00:17:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/967096462 ]] 2023-10-14T00:17:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/967096462 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\6\7\0\9\6\4\6\2 ]] 2023-10-14T00:17:18.373 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:17:18.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:18.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:18.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:18.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:18.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:18.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:18.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:18.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:18.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:18.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:18.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:18.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:18.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:18.968+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:18.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:19.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:19.009 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:19.008+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:19.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:19.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:19.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:19.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:19.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:19.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:19.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:19.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:19.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:19.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:19.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:19.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:19.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:19.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:19.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:19.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:19.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:19.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:20.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:20.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:20.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:20.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:20.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:20.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:20.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:20.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:21.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:21.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:21.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:21.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:21.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:21.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:21.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:21.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:21.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:21.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:21.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:21.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:21.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:21.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:21.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:23.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:23.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:23.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:23.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:23.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:23.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:23.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:23.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:23.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:23.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:24.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:24.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:24.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:24.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:24.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:24.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:24.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:24.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:24.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:24.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:24.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:24.360+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:24.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:24.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:24.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:24.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:24.379 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:24.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:25.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:25.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:25.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:25.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:25.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:25.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:25.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:25.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:26.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:17:26.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:26.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:26.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:26.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:26.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:26.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:26.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:26.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:27.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:27.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:27.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:27.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:27.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:27.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:27.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:27.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:28.375 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:17:28.382 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:28.398 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:28.405 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:28.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:28.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:28.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:28.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:28.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:28.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:28.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:28.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:29.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:29.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:29.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:29.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:29.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:29.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:29.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:29.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:29.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:29.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:29.831 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/967096462 2023-10-14T00:17:29.850 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:17:29.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242649 2023-10-14T00:17:29.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:29.852 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/967096462 ]] 2023-10-14T00:17:29.852 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/967096462 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\6\7\0\9\6\4\6\2 ]] 2023-10-14T00:17:29.852 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:17:30.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:30.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:30.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:30.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:30.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:30.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:31.573 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.572+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:31.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:31.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:31.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:31.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:31.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:31.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:31.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:31.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:31.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:31.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:31.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:31.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:31.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:31.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:31.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:31.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:17:31.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:31.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.840+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:31.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:31.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:31.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:31.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:31.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:32.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:32.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:32.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:32.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:32.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:32.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:32.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:32.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:33.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:33.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:33.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:33.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:35.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:35.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:35.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:35.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:35.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:35.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:35.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:35.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:35.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:35.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:35.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:35.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:36.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:36.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:17:36.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:36.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:36.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:36.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:36.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:36.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:36.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:36.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:36.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:36.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:37.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:37.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:37.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:37.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:37.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:37.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:37.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:37.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:37.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:37.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:38.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:38.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:38.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:38.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:38.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:38.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:38.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:38.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:39.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:39.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:39.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:39.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:39.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:39.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:39.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:39.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:39.863 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:17:39.866 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:39.866 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:39.882 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:41.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:17:41.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:41.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:41.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:41.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:41.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:41.319 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/762069660 2023-10-14T00:17:41.325 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:17:41.327 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242661 2023-10-14T00:17:41.327 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:41.327 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/762069660 ]] 2023-10-14T00:17:41.327 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/762069660 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\9\6\7\0\9\6\4\6\2 ]] 2023-10-14T00:17:41.327 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/762069660 2023-10-14T00:17:41.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:41.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:41.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:41.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:41.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:41.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:17:41.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:41.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:41.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:41.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:41.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:41.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:41.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:41.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:41.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:41.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:41.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:41.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:42.289 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:42.288+0000 7f0826b0e640 -1 librbd::image::GetMetadataRequest: 0x5627d859bec0 handle_metadata_list: failed to retrieve image metadata: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:42.289 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:42.288+0000 7f0826b0e640 -1 librbd::image::RefreshRequest: failed to retrieve metadata: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:42.290 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:42.288+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to refresh image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:42.290 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:17:42.288+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8aaf180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:43.231 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/762069660 until 2023-10-14T01:17:42.500272+0000 (3600 sec) 2023-10-14T00:17:43.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:43.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:43.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:43.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:43.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:43.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:43.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:43.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:43.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:43.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:43.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:43.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:43.288 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:17:43.294 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/762069660 2023-10-14T00:17:44.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:44.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:44.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:44.068+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:44.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:44.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:44.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:44.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:44.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:44.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:44.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:44.366 INFO:tasks.workunit.client.0.smithi138.stderr:listed 19 entries 2023-10-14T00:17:44.434 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/762069660 2023-10-14T00:17:44.434 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:17:44.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:44.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:44.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:44.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:44.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:44.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:44.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:44.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:44.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:44.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:44.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:45.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:45.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:17:45.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:45.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:45.485 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:45.484+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:45.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:45.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:45.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:45.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:45.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:45.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:45.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:45.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:48.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:48.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:48.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:48.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:48.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:48.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:48.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:48.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:48.604+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:17:48.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:48.676+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:17:48.695 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:48.696+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:17:48.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:48.704+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:17:48.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:48.704+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:17:48.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:17:48.704+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:17:48.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:48.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:48.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:48.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:48.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:17:48.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:48.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:48.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:48.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:48.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:49.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:49.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:17:49.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:17:49.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:49.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:49.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:49.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:49.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:49.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:49.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:49.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:50.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:50.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:17:51.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:51.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:51.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:51.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:51.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:51.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:51.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:51.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:51.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:51.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:51.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:51.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:53.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:53.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:53.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:53.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:53.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:53.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:53.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:53.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:54.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:54.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:54.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:54.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:54.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:54.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:54.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:54.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:54.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:54.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:54.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:54.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:54.436 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:17:54.440 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:54.443 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:54.453 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:55.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:55.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:55.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:55.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:55.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:55.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:55.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:55.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:55.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:55.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:55.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3135251554 2023-10-14T00:17:55.806 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:17:55.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242675 2023-10-14T00:17:55.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:55.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3135251554 ]] 2023-10-14T00:17:55.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3135251554 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\7\6\2\0\6\9\6\6\0 ]] 2023-10-14T00:17:55.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3135251554 2023-10-14T00:17:56.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:56.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:17:56.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:56.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:56.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:56.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:56.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:56.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:57.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:57.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:57.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:57.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:57.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:57.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:57.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:57.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:57.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:57.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:57.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:57.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:57.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:17:57.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:17:57.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:57.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:57.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:57.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:57.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:57.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:58.172 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3135251554 until 2023-10-14T01:17:57.427736+0000 (3600 sec) 2023-10-14T00:17:58.237 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3135251554 2023-10-14T00:17:58.258 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:17:58.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:58.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:17:58.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:58.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:17:58.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:58.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:17:58.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:58.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:17:58.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:17:58.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:17:59.382 INFO:tasks.workunit.client.0.smithi138.stderr:listed 20 entries 2023-10-14T00:17:59.437 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3135251554 2023-10-14T00:17:59.437 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:18:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.011+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.011+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.011+0000 7f082630d640 -1 librbd::ImageState: 0x5627d86cf100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.011+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7cc3600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.019+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.019+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.041 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.039+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.041 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.039+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7cc3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.047+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.048 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.047+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b19600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd7480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.075+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.075+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.079 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.079+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.079 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.079+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.103+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.103+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e17d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.107 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.107+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.107 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.107+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e17b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.111+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.111+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.116 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.115+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.116 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.115+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e17900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.127+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.127+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e17700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.147 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.147+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.148 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.147+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e17500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.148 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.147+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.148 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.147+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.158 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.159+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.159 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:18:00.159+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:01.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:01.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:01.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:01.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:01.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:01.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:01.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:01.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:01.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:01.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:01.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:01.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:03.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:03.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:03.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:03.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:03.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:03.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:03.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:03.491+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:03.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:03.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:03.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:03.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:03.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:03.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:03.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:03.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:03.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:03.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:03.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:03.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:03.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:04.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:04.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:04.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:04.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:06.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:06.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:06.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:06.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:06.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:06.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:06.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:06.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:06.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:06.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:06.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:06.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:07.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:07.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:18:07.027 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:07.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:07.035 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:07.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:07.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:07.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:07.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:07.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:07.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:07.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:18:07.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:07.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:07.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:07.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:07.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:07.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:07.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:07.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:07.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:07.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:07.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:07.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:08.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:08.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:08.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:08.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:08.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:08.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:08.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:08.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:18:08.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:08.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:08.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:08.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:08.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:08.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:08.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:09.440 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:09.441 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:09.441 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:09.441 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:18:10.726 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3135251554 2023-10-14T00:18:10.727 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:18:10.730 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242690 2023-10-14T00:18:10.730 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:10.730 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3135251554 ]] 2023-10-14T00:18:10.730 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3135251554 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\3\5\2\5\1\5\5\4 ]] 2023-10-14T00:18:10.731 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:18:12.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:12.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:12.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:12.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:12.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:12.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:12.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:12.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:12.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:12.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:12.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:12.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:12.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:12.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:12.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:14.425 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:14.423+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:18:14.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:14.423+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:14.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:14.443+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:14.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:14.443+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:14.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:14.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:14.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:14.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:18:14.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:14.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:16.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:18:16.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:16.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:16.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:16.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:16.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:16.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:16.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:16.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:16.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:16.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:16.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:16.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:16.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:16.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:16.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:16.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:16.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:16.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:16.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:16.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:16.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:16.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:18.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:18.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:18.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:18.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:18.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:18.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:20.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:20.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:20.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:20.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:20.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:20.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:20.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:20.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:20.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:20.734 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:18:20.739 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:20.743 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:20.754 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:20.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:20.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:20.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:20.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:20.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:20.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:20.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:20.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:21.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:21.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:21.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:21.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:21.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:21.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:21.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:21.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:21.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:21.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:21.864 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:21.863+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:21.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:21.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:21.886 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:21.883+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:21.886 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:21.883+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:18:21.886 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:21.883+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:21.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:21.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:22.121 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3135251554 2023-10-14T00:18:22.121 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:18:22.128 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242702 2023-10-14T00:18:22.128 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:22.129 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3135251554 ]] 2023-10-14T00:18:22.129 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3135251554 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\3\5\2\5\1\5\5\4 ]] 2023-10-14T00:18:22.129 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:18:22.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:22.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:22.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:22.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:22.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:22.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:22.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:22.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:22.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:22.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:23.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:18:23.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:23.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:23.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:23.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:23.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:23.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:23.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:23.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:23.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:23.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:23.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:23.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:23.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:24.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:24.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:24.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:24.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:24.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:24.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:24.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:24.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:24.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:24.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:24.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:24.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:25.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:25.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:25.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:25.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:25.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:25.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:25.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:25.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:25.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:25.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:25.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:25.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:25.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:25.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:25.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:25.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:25.730 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:18:25.731+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:18:26.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:26.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:26.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:26.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:26.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:26.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:26.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:26.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:27.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:27.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:27.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:27.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:27.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:27.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:28.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:28.099+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:28.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:28.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:29.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:29.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:29.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:29.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:29.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:29.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:29.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:29.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:29.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:29.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:29.501 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:29.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:29.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:29.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:29.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:29.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:29.568 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:29.567+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:29.568 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:29.567+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:18:29.569 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:29.567+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:30.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:30.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:30.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:30.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:30.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:30.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:30.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:30.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:30.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:30.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:30.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:30.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:30.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:30.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:30.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:30.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:30.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:31.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:31.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:31.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:31.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:32.130 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:18:32.130 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:33.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3135251554 2023-10-14T00:18:33.391 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:18:33.391 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242713 2023-10-14T00:18:33.391 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:33.392 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3135251554 ]] 2023-10-14T00:18:33.392 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3135251554 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\3\5\2\5\1\5\5\4 ]] 2023-10-14T00:18:33.392 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:18:34.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:34.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:34.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:34.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:34.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:34.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:34.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:34.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:34.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:34.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:34.992 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:34.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:34.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:34.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:34.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:34.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:18:34.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:34.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:35.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:35.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:35.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:35.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:35.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:35.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:35.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:35.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:35.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:35.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:35.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:35.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:36.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:18:36.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:18:36.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:36.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:36.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:36.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:36.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:36.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:36.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:36.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:36.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:18:36.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:36.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:36.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:36.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:36.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:36.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:36.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:36.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:37.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:37.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:37.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:37.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:37.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:37.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:37.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:37.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:37.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:37.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:38.648 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:38.647+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:18:38.648 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:38.647+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:38.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:38.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:38.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:38.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:38.689 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:38.687+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:38.689 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:38.687+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:18:38.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:38.687+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:39.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:18:39.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:39.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:39.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:39.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:39.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:39.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:39.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:39.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:40.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:40.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:40.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:40.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:40.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:40.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:40.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:40.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:40.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:40.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:40.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:40.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:40.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:43.397 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:43.397 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:18:43.406 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:43.411 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:44.691 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3135251554 2023-10-14T00:18:44.692 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:18:44.709 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242724 2023-10-14T00:18:44.710 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:44.710 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3135251554 ]] 2023-10-14T00:18:44.710 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3135251554 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\3\5\2\5\1\5\5\4 ]] 2023-10-14T00:18:44.710 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:18:48.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:48.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:48.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:48.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:48.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:48.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:48.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:48.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:48.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:48.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:48.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:48.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:48.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:48.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:51.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:51.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:51.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:51.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:51.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:51.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:51.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:51.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:51.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:51.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:51.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:51.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:51.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:52.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:52.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:52.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:52.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:18:52.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:52.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:52.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:52.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:52.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:52.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:52.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:52.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:53.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:53.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:18:53.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:53.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:53.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:53.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:53.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:53.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:53.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:53.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:53.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:53.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:53.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:53.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:53.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:53.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:18:53.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:53.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:18:53.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:53.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:18:53.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:53.234+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:18:53.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:53.234+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:18:53.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:18:53.234+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:18:54.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:54.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:54.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:54.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:54.712 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:54.712 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:18:54.713 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:54.725 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:54.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:54.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:54.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:54.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:54.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:54.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:54.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:56.140 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3179505995 2023-10-14T00:18:56.150 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:18:56.151 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242736 2023-10-14T00:18:56.151 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:56.151 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3179505995 ]] 2023-10-14T00:18:56.152 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3179505995 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\3\5\2\5\1\5\5\4 ]] 2023-10-14T00:18:56.152 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3179505995 2023-10-14T00:18:57.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:57.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:18:57.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:57.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:18:57.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:57.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:57.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:57.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:57.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:57.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:57.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:57.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:58.190 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3179505995 until 2023-10-14T01:18:57.511053+0000 (3600 sec) 2023-10-14T00:18:58.304 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:18:58.305 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3179505995 2023-10-14T00:18:59.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:59.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:18:59.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:59.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:18:59.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:59.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:18:59.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:18:59.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:18:59.505 INFO:tasks.workunit.client.0.smithi138.stderr:listed 21 entries 2023-10-14T00:18:59.558 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3179505995 2023-10-14T00:18:59.558 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:19:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.022+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.022+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.022+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8283480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.022+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e17480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.074+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.074+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.078+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.078+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8e16500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.090+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.090+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8e16e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.102+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.102+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.102+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.102+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8e17080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.102+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.102+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.120 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.118+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.121 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.118+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.125 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.122+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.126 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.122+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.130+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.130+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.130+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.130+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.138 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.138+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.139 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:19:00.138+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae3100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:00.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:00.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:00.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:00.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:00.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:00.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:00.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:00.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:00.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:00.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:00.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:00.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:00.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:00.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:00.774+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:00.806 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:00.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:00.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:00.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:00.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:00.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:00.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:00.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:01.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:01.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:01.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:01.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:01.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:01.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:01.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:01.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:01.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:01.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:02.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:02.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:02.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:02.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:02.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:02.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:02.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:02.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:02.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:02.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:03.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:03.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:03.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:03.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:03.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:03.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:03.345 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:03.342+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:03.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:03.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:03.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:03.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:03.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:03.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:04.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:04.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:04.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:04.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:04.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:04.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:04.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:04.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:04.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:04.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:04.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:04.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:05.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:05.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:05.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:05.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:05.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:05.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:05.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:05.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:05.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:05.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:05.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:05.838+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:05.840 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:05.838+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:05.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:05.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:05.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:05.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:05.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:05.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:06.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:06.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:06.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:06.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:06.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:06.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:06.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:06.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:06.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:06.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:06.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:06.514+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:06.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:06.518+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:06.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:06.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:06.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:06.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:06.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:06.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:06.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:06.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:06.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:06.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:07.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:07.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:07.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:07.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:07.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:07.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:07.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:07.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:07.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:07.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:07.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:07.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:07.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:07.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:08.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:08.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:08.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:08.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:08.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:08.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:08.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:08.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:09.562 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:09.566 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:09.567 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:19:09.567 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:09.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:09.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:09.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:09.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:09.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:09.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:10.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:10.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3179505995 2023-10-14T00:19:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:19:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242751 2023-10-14T00:19:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3179505995 ]] 2023-10-14T00:19:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3179505995 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\7\9\5\0\5\9\9\5 ]] 2023-10-14T00:19:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:19:12.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:12.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:12.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:12.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:12.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:12.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:12.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:12.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:12.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:12.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:12.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:12.834 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:12.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:12.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:12.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:12.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:12.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:13.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:13.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:13.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:13.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:13.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:13.514+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:13.535 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:13.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:13.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:13.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:13.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:13.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:13.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:13.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:14.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:14.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:14.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:14.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:14.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:14.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:14.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:14.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:14.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:14.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:14.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:14.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:14.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:14.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:15.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:15.714+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:15.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:15.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:15.733 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:15.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:15.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:15.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:15.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:15.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:16.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:16.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:16.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:16.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:16.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:16.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:16.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:16.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:16.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:16.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:16.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:16.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:17.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:17.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:17.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:17.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:17.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:17.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:19.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:19.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:19.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:19.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:19.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:19.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:19.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:19.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:19.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:19.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:20.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:20.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:20.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:20.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:20.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:20.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:20.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:20.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:20.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:20.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:21.344 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:21.345 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:19:21.378 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:21.387 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:21.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:21.406+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:21.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:21.450+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:21.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:21.450+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:21.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:21.450+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:22.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:22.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:22.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:22.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:22.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:22.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:22.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:22.564 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:22.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:22.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:22.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:22.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:22.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:22.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:22.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:22.840 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3179505995 2023-10-14T00:19:22.840 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:19:22.842 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242762 2023-10-14T00:19:22.843 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:22.843 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3179505995 ]] 2023-10-14T00:19:22.843 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3179505995 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\7\9\5\0\5\9\9\5 ]] 2023-10-14T00:19:22.843 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:19:23.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:23.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:23.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:23.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:23.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:23.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:23.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:23.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:23.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:23.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:23.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:23.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:23.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:23.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:23.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:24.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:24.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:26.134 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:26.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:26.134 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:26.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:26.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:26.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:26.207 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:26.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:26.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:26.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:26.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:26.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:26.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:26.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:26.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:26.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:26.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:26.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:26.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:26.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:26.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:26.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:26.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:26.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:26.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:26.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:26.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:26.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:26.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:26.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:26.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:26.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:26.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:27.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:27.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:27.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:27.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:27.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:27.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:27.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:27.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:19:27.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:19:27.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:27.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:27.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:27.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:27.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:27.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:27.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:28.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:28.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:28.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:28.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:28.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:28.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:28.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:28.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:28.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:28.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:28.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:28.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:28.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:28.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:28.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:28.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:28.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:28.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:28.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:28.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:28.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:28.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:28.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:28.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:30.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:30.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:30.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:30.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:30.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:30.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:30.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:30.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:30.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:30.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:30.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:30.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:30.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:30.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:31.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:31.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:32.856 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:32.856 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:32.857 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:32.861 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:19:33.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:33.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:33.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:33.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:33.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:33.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:33.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:33.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:33.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:33.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:33.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:33.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:33.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:34.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:19:34.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:34.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:34.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:34.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:34.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:34.169 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3179505995 2023-10-14T00:19:34.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:34.172 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:19:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242774 2023-10-14T00:19:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3179505995 ]] 2023-10-14T00:19:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3179505995 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\7\9\5\0\5\9\9\5 ]] 2023-10-14T00:19:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:19:34.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:34.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:34.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:35.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:35.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:35.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:35.430+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:35.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:35.430+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:35.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:35.430+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:35.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:35.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:35.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:35.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:35.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:35.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:36.259 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:36.260 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:36.268 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:36.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:36.268 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:36.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:36.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:36.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:36.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:36.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:37.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:19:37.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:19:37.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:37.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:37.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:37.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:37.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:37.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:37.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:37.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:37.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:39.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:39.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:39.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:39.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:39.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:39.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:39.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:39.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:39.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:39.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:39.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:39.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:40.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:40.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:40.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:40.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:40.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:40.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:40.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:40.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:40.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:40.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:41.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:40.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:43.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:43.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:43.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:43.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:43.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:43.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:43.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:43.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:43.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:43.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:43.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:43.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:43.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:43.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:43.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:43.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:43.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:43.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:43.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:43.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:43.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:43.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:43.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:43.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:43.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:43.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:43.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:43.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:43.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:44.195 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:44.197 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:19:44.218 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:44.234 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:44.726 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:44.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:44.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:44.733+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:44.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:44.741+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:44.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:44.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:44.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:44.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:44.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:44.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:45.628 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3179505995 2023-10-14T00:19:45.628 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:19:45.629 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242785 2023-10-14T00:19:45.629 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:45.629 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3179505995 ]] 2023-10-14T00:19:45.629 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3179505995 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\7\9\5\0\5\9\9\5 ]] 2023-10-14T00:19:45.629 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:19:45.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:45.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:45.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:45.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:45.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:45.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:45.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:45.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:45.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:45.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:45.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:45.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:45.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:45.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:46.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:46.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:46.672 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:46.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:46.672 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:46.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:46.673 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:46.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:46.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:46.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:46.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:46.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:46.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:46.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:46.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:46.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:47.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:47.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:47.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:47.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:47.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:47.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:47.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:47.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:47.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:50.357 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:50.349+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:50.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:50.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:50.414 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:50.409+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:50.425 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:50.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:50.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:50.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:50.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:50.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:51.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:51.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:19:51.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:51.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:51.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:51.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:51.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:51.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:51.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:51.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:51.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:51.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:51.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:51.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:52.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:52.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:52.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:52.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:52.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:52.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:52.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:52.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:52.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:52.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:52.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:52.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:53.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:53.777+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:53.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:53.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:53.899 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:19:53.897+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:19:55.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:55.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:55.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:19:55.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:55.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:19:55.469 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:55.465+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:55.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:55.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:55.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:55.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:55.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:55.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:19:55.631 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:55.632 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:55.639 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:55.647 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:19:55.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:19:55.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:19:55.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:19:55.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:19:55.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:19:55.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:19:57.076 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3179505995 2023-10-14T00:19:57.076 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:19:57.077 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242797 2023-10-14T00:19:57.077 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:57.077 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3179505995 ]] 2023-10-14T00:19:57.078 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3179505995 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\7\9\5\0\5\9\9\5 ]] 2023-10-14T00:19:57.078 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:19:59.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:59.609+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:19:59.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:59.609+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:19:59.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:59.621+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:19:59.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:59.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:19:59.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:59.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:19:59.657 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:59.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:19:59.657 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:19:59.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:00.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:00.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:00.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:00.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:00.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:00.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:00.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:00.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:01.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:01.157+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:01.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:01.173+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:01.185 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:01.185+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:01.194 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:01.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:01.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:01.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:02.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:02.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:02.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:02.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:02.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:02.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:02.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:02.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:02.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:02.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:02.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:02.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:02.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:02.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:02.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:02.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:02.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:02.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:02.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:02.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:02.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:02.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:05.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:05.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:05.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:05.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:05.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:05.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:05.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:05.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:05.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:05.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:05.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:05.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:06.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:06.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:20:06.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:06.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:06.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:06.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:06.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:06.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:06.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:06.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:06.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:06.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:06.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:06.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:07.080 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:07.080 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:20:07.081 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:07.083 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:07.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:07.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:07.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:07.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:07.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:07.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:07.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:07.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:07.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:07.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:07.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:07.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:08.501 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/327634990 2023-10-14T00:20:08.502 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:20:08.502 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242808 2023-10-14T00:20:08.503 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:08.503 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/327634990 ]] 2023-10-14T00:20:08.503 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/327634990 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\1\7\9\5\0\5\9\9\5 ]] 2023-10-14T00:20:08.503 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/327634990 2023-10-14T00:20:08.754 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:08.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:20:08.754 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:08.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:08.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:08.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:08.787 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:08.789+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:08.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:08.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:08.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:08.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:08.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:08.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:10.281 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.281+0000 7f0825b0c640 -1 librbd::mirror::GetInfoRequest: 0x5627d8211980 handle_get_snapshots: failed to get snapshots: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.290 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.293+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d364ccc0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.294 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.297+0000 7f0825b0c640 -1 librbd::mirror::GetInfoRequest: 0x5627d859a840 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.298 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.301+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8463980 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.303 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.305+0000 7f0825b0c640 -1 librbd::mirror::GetInfoRequest: 0x5627d8462a80 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.304 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.305+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8462b40 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.305 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.305+0000 7f0825b0c640 -1 librbd::mirror::GetInfoRequest: 0x5627d364cfc0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.305 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.309+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8c69440 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.306 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.309+0000 7f0825b0c640 -1 librbd::mirror::GetInfoRequest: 0x5627d859bec0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.314 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:20:10.313+0000 7f0823307640 -1 librbd::api::Mirror: image_info_list: failed to list mirror image statuses: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:10.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:10.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:10.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:10.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:10.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:10.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:10.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:10.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:10.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:10.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:10.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:10.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:10.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:10.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:10.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:10.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:10.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:10.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:10.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:10.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:10.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:10.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:10.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:10.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:11.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:11.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:11.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:11.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:11.265 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/327634990 until 2023-10-14T01:20:10.534947+0000 (3600 sec) 2023-10-14T00:20:11.324 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/327634990 2023-10-14T00:20:11.324 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:20:12.511 INFO:tasks.workunit.client.0.smithi138.stderr:listed 22 entries 2023-10-14T00:20:12.612 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/327634990 2023-10-14T00:20:12.612 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:20:12.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:12.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:12.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:12.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:12.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:12.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:13.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:13.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:20:13.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:13.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:13.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:13.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:13.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:13.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:13.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:13.129+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:13.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:13.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:13.149 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:13.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:13.160 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:13.161+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:13.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:13.161+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:13.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:13.161+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:14.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:14.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:14.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:14.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:14.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:14.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:14.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:14.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:14.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:14.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:14.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:14.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:15.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:15.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:20:15.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:15.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:15.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:15.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:15.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:15.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:15.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:15.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:15.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:15.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:15.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:15.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:19.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:19.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:19.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:19.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:19.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:19.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:19.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:19.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:19.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:19.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:19.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:19.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:20.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:20.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:20.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:20.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:20:20.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:20.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:20.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:20.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:20.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:20.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:20.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:20.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:22.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:20:22.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:22.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:22.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:22.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:22.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:22.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:22.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:22.620 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:20:22.621 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:22.621 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:22.623 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:22.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:20:22.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:20:22.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:22.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:22.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:22.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:22.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:22.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:22.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:22.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:22.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:22.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:22.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:22.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:22.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:22.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:23.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:23.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:23.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:23.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:23.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:23.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:23.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:23.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:23.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:23.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:23.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:23.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:23.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:24.080 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3463294261 2023-10-14T00:20:24.083 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:20:24.086 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242824 2023-10-14T00:20:24.087 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:24.087 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3463294261 ]] 2023-10-14T00:20:24.087 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3463294261 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\2\7\6\3\4\9\9\0 ]] 2023-10-14T00:20:24.087 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3463294261 2023-10-14T00:20:24.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:24.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:24.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:24.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:24.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:24.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:24.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:24.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:24.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:24.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:25.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:25.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:20:25.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:25.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:25.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:25.469+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:25.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:25.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:25.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:25.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:25.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:25.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:25.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:25.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:26.209 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3463294261 until 2023-10-14T01:20:25.537832+0000 (3600 sec) 2023-10-14T00:20:26.248 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:20:26.274 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3463294261 2023-10-14T00:20:26.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:26.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:26.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:26.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:26.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:26.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:26.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:26.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:26.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:26.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:26.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:26.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:26.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:26.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:27.361 INFO:tasks.workunit.client.0.smithi138.stderr:listed 23 entries 2023-10-14T00:20:27.407 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3463294261 2023-10-14T00:20:27.407 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:20:27.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:27.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:27.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:27.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:27.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:27.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:27.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:27.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:27.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:27.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:27.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:27.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:28.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:28.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:28.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:28.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:28.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:28.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:28.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:28.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:28.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:28.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:28.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:28.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:29.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:29.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:29.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:29.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:29.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:29.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:29.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:29.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:29.331 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:29.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:32.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:20:32.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:20:32.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:32.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:32.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:32.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:32.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:32.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:32.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:32.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:32.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:32.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:32.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:32.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:32.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:32.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:32.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:32.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:32.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:32.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:33.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:33.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:33.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:33.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:20:33.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:33.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:33.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:33.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:33.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:33.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:33.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:34.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:34.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:35.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:35.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:35.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:35.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:35.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:35.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:35.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:35.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:35.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:35.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:36.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:35.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:36.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:36.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:36.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:36.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:36.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:36.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:36.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:36.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:36.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:36.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:37.415 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:37.420 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:20:37.429 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:37.434 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:37.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:37.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:37.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:37.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:37.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:37.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:37.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:37.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:37.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:37.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:37.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:37.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:38.783 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3463294261 2023-10-14T00:20:38.783 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:20:38.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242838 2023-10-14T00:20:38.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:38.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3463294261 ]] 2023-10-14T00:20:38.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3463294261 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\4\6\3\2\9\4\2\6\1 ]] 2023-10-14T00:20:38.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:20:40.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:40.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:40.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:40.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:40.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:40.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:40.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:40.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:40.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:40.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:40.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:40.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:40.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:40.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:40.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:40.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:40.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:40.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:40.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:40.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:40.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:40.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:40.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:40.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:43.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:43.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:43.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:43.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:43.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:43.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:43.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:43.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:43.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:43.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:45.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:45.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:45.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:45.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:45.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:45.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:45.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:45.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:45.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:45.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:45.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:45.744+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:46.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:46.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:46.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:46.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:46.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:46.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:47.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:47.452+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:20:47.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:47.452+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:47.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:47.460+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:47.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:47.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:47.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:47.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:47.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:47.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:47.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:47.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:48.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:48.060+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:48.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:48.060+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:48.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:48.060+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:48.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:20:48.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:48.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:48.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:48.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:48.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:48.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:48.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:48.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:48.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:20:48.810 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:48.811 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:48.817 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:49.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:49.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:49.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:49.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:49.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:49.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:50.255 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3463294261 2023-10-14T00:20:50.255 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:20:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242850 2023-10-14T00:20:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3463294261 ]] 2023-10-14T00:20:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3463294261 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\4\6\3\2\9\4\2\6\1 ]] 2023-10-14T00:20:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:20:50.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:50.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:20:50.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:50.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:50.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:50.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:20:50.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:50.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:50.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:50.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:50.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:50.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:50.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:50.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:51.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:51.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:51.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:51.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:51.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:51.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:51.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:51.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:51.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:51.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:51.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:51.840+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:51.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:51.840+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:53.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:53.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:53.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:53.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:53.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:53.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:54.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:54.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:54.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:54.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:54.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:54.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:54.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:54.196+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:54.196 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:54.196+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:54.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:54.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:54.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:54.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:54.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:54.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:54.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:54.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:54.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:54.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:54.976 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:54.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:54.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:54.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:55.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:55.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:55.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:55.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:55.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:55.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:56.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:56.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:56.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:56.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:56.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:56.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:20:56.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:56.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:56.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:56.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:56.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:56.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:56.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:56.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:56.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:56.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:56.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:56.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:20:56.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:56.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:56.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:56.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:56.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:57.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:57.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:20:57.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:57.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:20:57.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:57.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:20:57.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:57.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:20:57.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:20:57.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:20:57.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:57.712+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:57.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:57.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:57.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:57.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:57.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:57.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:58.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.300+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:20:58.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.300+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:20:58.303 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:20:58.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:20:58.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:20:58.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:20:58.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:20:58.495 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:20:58.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.000+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.000+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.004+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.004+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd7580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.009 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.008+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.009 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.008+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8282900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.058 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::ImageState: 0x5627d7ae7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.058 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8deaa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.058 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.059 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.056+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b18f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.060+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.060+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.060+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8861780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.060+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae2700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.060+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.060+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.068+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.068+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae2f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.068+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.068+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.072+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.072+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.083 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.080+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.084 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.080+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.092+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.092+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.092+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.092+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.113 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.112+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.112+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.112+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.112+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.112+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.112+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.128+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:21:00.128+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.257 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:21:00.257 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:00.261 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:00.274 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:00.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:00.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:00.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:00.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:00.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:02.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:21:02.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:02.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:02.146 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3463294261 2023-10-14T00:21:02.148 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:21:02.149 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242862 2023-10-14T00:21:02.149 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:02.149 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3463294261 ]] 2023-10-14T00:21:02.150 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3463294261 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\4\6\3\2\9\4\2\6\1 ]] 2023-10-14T00:21:02.150 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:21:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:02.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:02.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:02.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:02.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:02.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:02.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:02.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:02.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:02.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:02.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:02.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:02.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:02.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:02.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:02.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:02.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:03.009 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.007+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:03.015 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:03.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:03.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:03.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:03.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:03.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:03.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:03.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:03.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:03.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:03.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:03.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:03.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:03.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:03.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:03.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:03.928 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.927+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:03.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.927+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:03.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:03.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.955+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:03.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:03.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:03.971 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:03.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:04.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:21:04.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:21:04.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:04.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:04.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:04.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:04.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:04.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:04.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:04.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:04.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:04.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:04.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:05.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:05.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:05.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:05.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:05.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:05.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:05.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:05.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:05.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:06.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:06.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:06.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:06.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:06.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:06.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:06.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:06.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:06.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:06.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:06.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:06.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:06.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:06.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:06.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:06.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:06.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:06.811+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:06.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:06.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:06.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:06.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:06.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:06.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:07.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:07.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:07.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:07.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:07.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:07.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:07.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:07.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:07.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:07.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:07.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:07.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:09.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:09.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:09.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:09.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:09.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:09.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:09.527 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:09.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:09.559 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:09.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:09.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:09.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:09.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:09.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:12.152 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:21:12.153 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:12.154 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:12.162 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:13.384 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3463294261 2023-10-14T00:21:13.385 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:21:13.385 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242873 2023-10-14T00:21:13.385 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:13.386 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3463294261 ]] 2023-10-14T00:21:13.386 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3463294261 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\4\6\3\2\9\4\2\6\1 ]] 2023-10-14T00:21:13.386 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:21:13.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:13.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:13.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:13.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:13.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:13.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:13.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:13.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:13.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:13.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:13.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:13.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:13.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:13.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:14.696 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:21:14.695+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:21:15.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:15.229 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:15.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:15.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.251+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:15.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.251+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:15.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.251+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:15.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:15.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:15.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:15.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:15.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:15.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:15.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:15.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:15.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:15.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:15.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:15.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:15.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:15.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:15.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:15.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:16.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:16.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:16.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:16.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:16.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:16.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:16.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:16.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:16.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:16.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.795+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:21:16.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:16.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:16.877 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.871+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:16.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.875+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:16.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.879+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:16.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.879+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:16.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:16.879+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:17.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:17.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:21:17.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:17.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:17.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:17.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:17.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:17.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:17.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:17.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:18.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:18.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:18.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:18.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:18.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:18.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:18.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:18.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:20.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:20.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:20.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:20.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:20.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:20.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:20.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:20.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:20.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:20.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:20.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:20.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:20.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:20.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:20.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:20.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:21.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:21.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:22.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:21:22.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:21:22.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:22.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:22.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:22.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:22.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:22.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:22.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:22.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:22.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:23.394 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:23.400 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:23.407 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:23.410 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:21:23.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:23.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:23.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:23.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:23.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:23.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:23.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:23.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:23.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:23.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:23.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:23.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:23.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:23.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:23.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:23.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:24.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:24.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:24.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:24.667 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:24.673 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.671+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:24.707 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:24.707 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:24.707 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:24.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:24.805 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3820507230 2023-10-14T00:21:24.806 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:21:24.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242884 2023-10-14T00:21:24.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:24.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3820507230 ]] 2023-10-14T00:21:24.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3820507230 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\4\6\3\2\9\4\2\6\1 ]] 2023-10-14T00:21:24.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3820507230 2023-10-14T00:21:25.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:25.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:25.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:25.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:25.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:25.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:25.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:25.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:25.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:25.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:25.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:25.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:25.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:25.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:25.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:25.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:25.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:25.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:25.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:25.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:25.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:25.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:25.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:26.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:26.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:26.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:26.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:26.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:26.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:26.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:26.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:26.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:26.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:26.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:26.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:26.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:26.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:26.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:26.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:26.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:26.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:26.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:26.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:26.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:26.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:27.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:27.031+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:27.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:27.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:27.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:27.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:27.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:27.091+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:27.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:27.091+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:27.093 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:27.091+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3820507230 until 2023-10-14T01:21:26.643923+0000 (3600 sec) 2023-10-14T00:21:27.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:21:27.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3820507230 2023-10-14T00:21:28.394 INFO:tasks.workunit.client.0.smithi138.stderr:listed 24 entries 2023-10-14T00:21:28.441 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3820507230 2023-10-14T00:21:28.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:21:28.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:28.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:28.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:28.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:28.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:28.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:29.033 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:21:29.031+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:21:29.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:29.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:29.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:29.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:29.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:29.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:29.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:29.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:29.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:29.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:29.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:29.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:29.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:29.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:29.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:30.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:30.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:30.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:30.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:30.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:30.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:30.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:30.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:30.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:30.903+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:30.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:30.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:30.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:30.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:30.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:30.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:30.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:30.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:31.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:31.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:31.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:31.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:31.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:31.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:31.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:31.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:31.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:31.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:31.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:31.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:31.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:31.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:31.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:31.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.779+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:31.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:31.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:31.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:31.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:31.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:31.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:31.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.955+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:31.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:31.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:31.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:31.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:33.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:33.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:33.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:33.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:33.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:33.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:33.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:33.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:34.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:34.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:34.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:34.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:34.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:34.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:34.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:34.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:34.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:34.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:35.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:35.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:35.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:35.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:35.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:35.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:36.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:36.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:36.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:36.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:36.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:36.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:36.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:36.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:36.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:36.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:36.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:36.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:36.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:36.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:36.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:36.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:36.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:37.272 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:37.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:37.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:37.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:37.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:37.319+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:37.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:37.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:37.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:37.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:37.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:37.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:37.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:37.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:37.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:37.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:37.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:37.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:38.443 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:38.447 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:38.463 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:38.467 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:21:38.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:38.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:38.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:38.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:38.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:38.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:38.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:38.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:38.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:38.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:38.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:38.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:38.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:38.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:39.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:39.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:39.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:39.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:39.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:39.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:21:39.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:39.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:39.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:39.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:39.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:39.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:39.865 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3820507230 2023-10-14T00:21:39.865 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:21:39.866 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242899 2023-10-14T00:21:39.866 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:39.866 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3820507230 ]] 2023-10-14T00:21:39.866 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3820507230 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\2\0\5\0\7\2\3\0 ]] 2023-10-14T00:21:39.867 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:21:40.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:21:40.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:40.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:40.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:40.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:40.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:40.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:40.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:40.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:40.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:40.494+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:40.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:40.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:40.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:40.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:40.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:40.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:41.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:41.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:41.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:41.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:41.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:41.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:41.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:41.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:41.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:41.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:41.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:41.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:41.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:41.609 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:41.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:41.609 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:41.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:41.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:41.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:41.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:41.630+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:41.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:41.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:42.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:42.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:42.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:42.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:42.185 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:42.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:42.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:42.216 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:42.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:42.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:42.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:42.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:42.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:42.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:42.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:42.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:42.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:42.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:42.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:42.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:42.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:42.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:21:42.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:42.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:42.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:42.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:42.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:42.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:42.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:42.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:42.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:42.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:42.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:42.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:43.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:43.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:43.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:43.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:43.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:43.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:43.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:43.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:44.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:44.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:44.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:44.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:44.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:44.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:44.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:44.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:44.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:44.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:45.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:45.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:45.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:45.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:45.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:45.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:45.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:45.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:45.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:45.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:45.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:45.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:45.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:45.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:45.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:45.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:45.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:45.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:45.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:45.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:45.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:45.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:45.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:45.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:45.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:46.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:46.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:46.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:46.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:46.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:46.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:46.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:46.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:47.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:47.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:47.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:47.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:47.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:47.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:47.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:47.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:47.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:47.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:47.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:47.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:47.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:47.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:21:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:49.874 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:49.884 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:50.230 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:50.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:50.230 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:50.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:50.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:50.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:50.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:50.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:50.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:50.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:50.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:50.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:50.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:50.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:51.367 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3820507230 2023-10-14T00:21:51.367 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:21:51.368 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242911 2023-10-14T00:21:51.368 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:51.368 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3820507230 ]] 2023-10-14T00:21:51.368 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3820507230 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\2\0\5\0\7\2\3\0 ]] 2023-10-14T00:21:51.369 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:21:53.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:53.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:53.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:53.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:55.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:55.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:55.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:55.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:55.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:55.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:55.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:55.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:55.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:21:56.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:21:56.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:56.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:21:56.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:56.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:56.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:56.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:56.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:56.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:56.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:56.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:21:56.566 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:56.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:56.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:56.638+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:56.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:56.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:56.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:56.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:56.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:56.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:56.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:56.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:58.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:58.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:58.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:58.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:58.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:21:58.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:21:58.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:21:58.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:21:58.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:21:58.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:21:58.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:21:59.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:59.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:21:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:59.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:21:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:59.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:21:59.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:59.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:21:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:59.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:21:59.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:21:59.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:00.016 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.014+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.017 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.014+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8283480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.017 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.014+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.018 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.014+0000 7f082630d640 -1 librbd::ImageState: 0x5627d6295500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.018 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.014+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.018 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.014+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.026+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.026+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.026+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.026+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.034 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.030+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.034 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.030+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.038+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.038+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.038+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.038+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.046 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.042+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.047 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.042+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.050+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.050+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.083 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.082+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.083 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.082+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.083 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.082+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.084 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.082+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.084 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.082+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.084 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.082+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.102+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.102+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.107 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.106+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.110+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.110+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.119 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.118+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:00.120 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:22:00.118+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:22:01.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:01.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:01.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:01.366+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:01.371 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:01.372 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:01.378 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:01.383 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:22:01.609 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:22:01.606+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:22:02.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.014+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:02.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:02.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:02.060 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:02.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:02.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:02.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.234+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:02.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:02.714 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3820507230 2023-10-14T00:22:02.718 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:22:02.719 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242922 2023-10-14T00:22:02.720 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:02.720 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3820507230 ]] 2023-10-14T00:22:02.720 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3820507230 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\2\0\5\0\7\2\3\0 ]] 2023-10-14T00:22:02.720 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:22:02.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:02.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:02.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:02.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:02.994+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:03.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:03.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:03.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:03.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:03.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:03.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:03.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:03.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:03.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:03.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:03.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:03.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:03.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:03.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:03.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:03.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:04.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:04.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:04.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:04.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:04.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:04.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:04.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:04.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:04.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:04.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:04.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:04.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:04.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:04.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:04.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:04.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:04.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:04.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:04.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:04.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:04.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:04.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:04.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:04.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:04.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:04.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:04.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:04.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:05.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:05.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:05.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:05.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:05.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:05.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:05.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:05.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:05.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:05.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:05.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:05.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:05.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:06.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:06.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:06.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:06.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:06.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:06.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:06.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:06.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:06.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:07.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:07.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:07.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:07.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:07.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:07.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:11.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:11.166+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:11.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:11.166+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:11.182 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:11.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:11.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:11.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:11.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:11.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:11.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:11.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:11.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:11.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:12.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:12.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:12.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:12.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:12.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:12.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:12.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:12.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:12.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:12.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:12.478 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:12.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:12.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:12.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:12.729 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:22:12.729 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:12.729 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:12.730 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:13.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:13.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:13.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:13.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:13.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:13.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:13.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:13.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:13.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:13.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:13.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:13.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:14.028 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3820507230 2023-10-14T00:22:14.029 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:22:14.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242934 2023-10-14T00:22:14.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:14.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3820507230 ]] 2023-10-14T00:22:14.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3820507230 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\2\0\5\0\7\2\3\0 ]] 2023-10-14T00:22:14.031 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:22:15.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:15.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:15.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:15.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:15.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:15.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:15.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:15.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:15.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:15.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:15.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:15.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:15.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:15.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:15.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:17.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:17.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:17.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:17.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:17.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:17.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:17.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:17.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:17.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:19.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:19.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:19.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:19.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:19.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:19.973+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:20.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:20.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:20.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:20.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:20.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:20.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:20.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:20.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:20.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:20.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:20.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:21.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:22:21.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:22:21.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:21.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:21.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:21.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:21.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:21.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:21.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:21.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:21.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:23.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:23.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:23.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:23.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:23.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:23.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:23.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:23.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:23.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:23.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:24.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:24.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:22:24.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:22:24.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:24.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:24.034 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:22:24.039 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:24.042 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:24.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:24.048 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:24.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:24.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:24.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:24.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:24.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:24.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:24.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:25.371 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3820507230 2023-10-14T00:22:25.372 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:22:25.375 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242945 2023-10-14T00:22:25.376 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:25.376 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3820507230 ]] 2023-10-14T00:22:25.376 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3820507230 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\2\0\5\0\7\2\3\0 ]] 2023-10-14T00:22:25.376 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:22:25.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:25.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:25.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:25.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:25.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:25.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:25.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:25.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:25.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:25.973+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:25.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:25.973+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:25.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:25.973+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:28.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:28.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:28.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:28.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:28.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:28.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:28.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:28.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:28.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:28.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:28.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:28.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:28.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:28.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:28.393 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:28.389+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:28.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:28.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:28.424 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:28.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:28.424 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:28.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:28.424 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:28.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:30.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:30.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:30.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:30.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:30.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:30.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:30.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:30.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:30.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:30.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:30.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:30.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:30.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:30.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:31.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:31.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:31.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:31.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:31.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:31.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:31.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:31.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:31.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:31.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:31.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:31.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:32.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:32.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:32.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:32.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:33.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:33.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:33.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:33.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:33.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:33.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:34.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:34.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:34.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:34.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:34.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:34.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:34.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:34.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:34.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:35.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:35.237+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:35.261 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:35.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:35.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:35.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:35.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:35.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:35.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:35.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:35.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:35.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:35.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:35.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:35.382 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:35.382 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:22:35.383 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:35.387 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:36.948 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2731291358 2023-10-14T00:22:36.948 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:22:36.950 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242956 2023-10-14T00:22:36.950 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:36.951 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2731291358 ]] 2023-10-14T00:22:36.951 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2731291358 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\2\0\5\0\7\2\3\0 ]] 2023-10-14T00:22:36.951 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2731291358 2023-10-14T00:22:37.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:37.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:37.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:37.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:37.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:37.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:37.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:37.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:37.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:38.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:38.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:38.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:38.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:38.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:38.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:38.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:38.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:38.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:38.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:38.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:38.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:38.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:38.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:38.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:38.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:38.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:38.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:38.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:38.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:38.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:38.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:39.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:39.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:39.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:39.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:39.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:39.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:39.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:39.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:39.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:39.294 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2731291358 until 2023-10-14T01:22:38.684802+0000 (3600 sec) 2023-10-14T00:22:39.369 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:22:39.376 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2731291358 2023-10-14T00:22:40.512 INFO:tasks.workunit.client.0.smithi138.stderr:listed 25 entries 2023-10-14T00:22:40.541 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2731291358 2023-10-14T00:22:40.542 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:22:40.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:40.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:40.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:40.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:40.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:40.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:22:40.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:22:40.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:40.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:40.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:40.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:40.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:40.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:40.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:40.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:40.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:41.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:41.133+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:41.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:41.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:41.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:41.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:42.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:42.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:42.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:42.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:42.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:42.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:42.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:42.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:42.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:42.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:42.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:42.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:42.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:42.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:42.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:42.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:42.842 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:42.841+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:42.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:42.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:42.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:42.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:42.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:42.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:44.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:44.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:44.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:44.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:44.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:44.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:44.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:44.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:44.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:44.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:44.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:45.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:45.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:45.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:45.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:45.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:45.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:45.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:45.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:45.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:45.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:45.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:45.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:45.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:45.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:45.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:45.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:45.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:45.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:46.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:46.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:46.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:46.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:46.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:46.533+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:46.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:46.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:46.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:46.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:46.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:46.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:46.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:46.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:47.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:47.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:47.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:47.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:47.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:47.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:47.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:47.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:48.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:48.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:48.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:48.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:48.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:48.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:48.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:48.593+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:48.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:48.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:48.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:48.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:49.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:49.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:49.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:49.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:49.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:49.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:49.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:49.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:49.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:49.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:49.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:49.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:49.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:49.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:50.544 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:50.544 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:22:50.545 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:50.563 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:51.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:22:51.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:22:51.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:51.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.945+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:51.949 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2731291358 2023-10-14T00:22:51.949 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:22:51.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242971 2023-10-14T00:22:51.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:51.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2731291358 ]] 2023-10-14T00:22:51.957 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2731291358 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\7\3\1\2\9\1\3\5\8 ]] 2023-10-14T00:22:51.957 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:22:51.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:51.986 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:51.986 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:51.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:51.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:54.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:54.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:54.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:54.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:54.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:54.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:54.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:54.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:54.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:54.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:54.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:54.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:55.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:55.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.153+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:55.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.157+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:55.200 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.201+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:55.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.201+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:55.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:55.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:55.908 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:55.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:55.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:55.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:55.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:22:55.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:55.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:55.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:55.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:55.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:55.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:55.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:55.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:56.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:56.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:56.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:56.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:56.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:56.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:56.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:56.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:56.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:56.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:57.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:57.201+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:57.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:57.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:22:57.227 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:57.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:57.250 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:57.249+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:22:57.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:57.249+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:22:57.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:57.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:22:58.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:22:58.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:58.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:58.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:58.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:22:58.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:22:58.695 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:58.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:22:58.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:22:58.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:22:58.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:22:58.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:22:58.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:22:58.729+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:22:58.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:22:58.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.020+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.020+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.020+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.020+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae2780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.020+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d81a9f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.020+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.040+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.040+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.052+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.052+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.065 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.064+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.065 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.064+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c0000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.065 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.064+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.066 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.064+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.066 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.064+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c0200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.066 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.064+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c0400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c0600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c0800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c0a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.076+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c0c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.092+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.092+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c1400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.096+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.096+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.096+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.096+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c1280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.096+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c1080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.098 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.096+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c1b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.108+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.108+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c1a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.108+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:23:00.108+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c1800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:01.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:01.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:01.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:01.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:01.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:01.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:01.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:01.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:01.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:01.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:01.984 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:23:01.986 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:01.987 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:01.988 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:02.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:23:02.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:02.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:02.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:02.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:02.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:02.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:02.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:02.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:03.400 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2731291358 2023-10-14T00:23:03.401 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:23:03.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242983 2023-10-14T00:23:03.402 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:03.402 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2731291358 ]] 2023-10-14T00:23:03.402 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2731291358 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\7\3\1\2\9\1\3\5\8 ]] 2023-10-14T00:23:03.402 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:23:03.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:03.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:03.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:03.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:03.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:03.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:03.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:03.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:03.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:03.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:04.791 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:04.788+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:04.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:04.788+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:04.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:04.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:04.837 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:04.836+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:04.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:04.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:04.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:04.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:04.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:04.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:05.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:05.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:05.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:05.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:05.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:05.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:05.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:05.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:05.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:05.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:05.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:05.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:05.227 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:05.228+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:06.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:06.132+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:06.160 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:06.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:06.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:06.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:06.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:06.192+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:08.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:08.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:08.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:08.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:08.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:08.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:08.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:08.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:08.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:09.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:09.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:09.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:09.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:09.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:09.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:09.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:09.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:09.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:09.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:11.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:11.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:11.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:11.192+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:11.213 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:11.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:11.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:11.216+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:11.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:11.264+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:11.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:11.264+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:12.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:12.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:12.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:12.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:12.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:12.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:12.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:12.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:12.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:12.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:13.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:13.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:13.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:13.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:13.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:13.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:13.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:13.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:13.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:13.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:13.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:13.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:13.403 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:13.403 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:23:13.405 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:13.420 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:14.411 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:23:14.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:14.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:14.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:14.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:14.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:14.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:14.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:14.547 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2731291358 2023-10-14T00:23:14.547 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:23:14.547 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697242994 2023-10-14T00:23:14.548 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:14.548 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2731291358 ]] 2023-10-14T00:23:14.548 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2731291358 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\7\3\1\2\9\1\3\5\8 ]] 2023-10-14T00:23:14.548 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:23:14.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.600+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:14.650 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:14.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.676+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:14.677 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:14.676+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:16.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:16.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:16.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:16.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:16.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:16.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:16.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:16.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:16.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:16.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:16.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:16.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:17.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:17.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:17.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:17.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:17.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:17.432 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:17.432 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:17.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:23:17.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:23:17.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:17.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:17.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:17.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:17.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:17.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:17.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:17.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:17.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:17.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:17.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:17.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:17.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:17.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:17.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:17.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:17.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:17.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:17.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:18.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:18.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:18.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:21.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:21.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:21.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:21.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:21.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:21.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:21.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:21.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:21.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:21.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:22.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:22.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:22.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:22.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:22.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:22.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:23.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:23.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:23.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:23.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:23.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:23.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:23.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:23.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:23.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:23.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:24.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:24.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:24.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:24.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:24.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:24.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:24.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:24.571 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:23:24.573 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:24.574 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:24.575 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:24.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:24.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:24.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:24.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:24.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:24.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:24.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:25.912 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2731291358 2023-10-14T00:23:25.912 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:23:25.918 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243005 2023-10-14T00:23:25.918 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:25.918 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2731291358 ]] 2023-10-14T00:23:25.919 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2731291358 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\7\3\1\2\9\1\3\5\8 ]] 2023-10-14T00:23:25.919 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:23:26.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:26.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:26.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:26.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:26.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:26.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:26.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:26.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:26.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:26.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:26.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:26.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:27.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:27.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:27.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:27.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:27.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:27.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:27.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:27.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:28.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:28.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:28.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:28.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:28.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:28.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:30.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:30.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:30.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:30.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:30.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:30.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:30.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:30.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:30.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:30.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:32.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:32.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:23:32.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:32.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:32.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:32.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:32.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:32.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:32.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:32.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:32.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:32.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:32.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:32.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:33.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:33.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:33.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:33.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:33.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:33.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:33.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:33.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:33.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:33.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:33.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:33.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:33.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:34.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:34.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:34.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:34.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:34.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:34.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:34.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:34.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:34.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:34.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:34.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:34.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:34.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:34.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:34.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:34.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:34.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:34.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:34.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:34.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:34.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:34.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:35.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:35.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:35.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:35.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:35.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:35.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:35.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:35.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:35.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:35.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:35.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:35.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:35.944 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:35.945 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:23:35.948 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:35.953 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:36.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:36.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:36.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:36.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:36.607 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:36.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:36.608 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:36.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:36.608 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:36.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:37.210 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2731291358 2023-10-14T00:23:37.211 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:23:37.211 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243017 2023-10-14T00:23:37.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:37.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2731291358 ]] 2023-10-14T00:23:37.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2731291358 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\7\3\1\2\9\1\3\5\8 ]] 2023-10-14T00:23:37.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:23:37.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:37.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:37.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:37.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:37.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:37.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:37.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:37.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:37.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:37.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:37.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:37.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:37.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:37.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:39.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:39.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:39.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:39.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:39.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:39.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:39.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:39.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:39.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:39.650 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:39.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:39.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:39.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:39.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:39.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:39.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:39.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:40.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:40.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:40.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:40.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:40.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:40.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:40.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:40.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:40.246 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:40.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:40.246 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:40.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:41.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:41.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:41.185 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:41.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:41.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:41.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:41.217 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:41.216+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:41.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:41.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:41.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:41.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:42.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:42.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:42.322 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:42.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:42.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:42.328+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:42.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:42.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:42.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:42.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:42.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:42.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:42.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:42.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:42.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:42.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:42.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:42.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:42.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:42.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:42.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:42.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:42.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:42.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:42.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:42.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:43.043 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:43.043 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:43.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:43.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:43.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:43.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:43.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:43.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:43.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:43.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:43.326 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:43.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:43.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:43.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:43.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:43.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:43.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:43.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:43.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:43.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:43.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:43.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:43.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:43.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:43.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:43.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:43.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:44.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:44.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:44.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:44.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:44.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:44.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:44.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:44.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:44.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:45.217 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:45.216+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:45.217 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:45.216+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:45.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:45.216+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:45.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:45.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:45.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:45.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:45.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:45.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:45.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:45.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:45.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:45.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:45.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:45.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:45.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:45.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:46.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:46.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:46.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:46.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:46.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:46.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:46.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:46.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:46.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:46.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:46.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:47.219 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:23:47.219 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:47.222 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:47.223 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:47.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:47.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:47.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:47.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:47.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:47.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:47.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:47.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:47.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:47.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:48.670 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2336971439 2023-10-14T00:23:48.675 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:23:48.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243028 2023-10-14T00:23:48.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:48.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2336971439 ]] 2023-10-14T00:23:48.677 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2336971439 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\7\3\1\2\9\1\3\5\8 ]] 2023-10-14T00:23:48.677 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2336971439 2023-10-14T00:23:48.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:48.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:49.873 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:49.871+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:49.873 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:49.871+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:49.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:49.887+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:49.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:49.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:49.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:49.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:49.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:49.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:49.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:49.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:50.124 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:23:50.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.127+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:23:50.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:50.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:50.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:50.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:50.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:50.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:50.143 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:50.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:51.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:51.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:51.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:51.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:51.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:51.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:51.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:51.354 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2336971439 until 2023-10-14T01:23:50.584007+0000 (3600 sec) 2023-10-14T00:23:51.418 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2336971439 2023-10-14T00:23:51.423 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:23:51.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:23:51.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:51.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:51.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:51.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:51.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:51.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:51.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:51.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:51.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:51.615+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:51.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:51.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:51.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:51.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:52.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:52.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:52.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:52.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:52.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:52.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:52.298 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:23:52.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:52.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:52.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:52.358 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.355+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:52.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:52.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:52.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:52.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:52.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:52.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.543+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:52.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.543+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:52.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.551+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:52.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.551+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:52.567 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:52.567 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:52.567 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:52.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:52.684 INFO:tasks.workunit.client.0.smithi138.stderr:listed 26 entries 2023-10-14T00:23:52.745 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2336971439 2023-10-14T00:23:52.745 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:23:53.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:53.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:53.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:53.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:53.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:53.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:53.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:53.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:53.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:53.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:53.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:53.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:53.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:53.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:53.755 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:53.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:53.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:53.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:56.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:56.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:56.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:56.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:56.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:56.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:56.392 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:23:56.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:56.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:56.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:56.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:56.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:56.410 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:56.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.587+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:23:56.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:56.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:23:56.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:56.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:56.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:56.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:23:57.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:57.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:23:57.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:57.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:57.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:57.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:57.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:57.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:57.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:57.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:57.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:57.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:57.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:57.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:57.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:58.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:58.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:23:58.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:58.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:58.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:58.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:23:58.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:23:58.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:23:58.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:58.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:58.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:58.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:59.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:59.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:23:59.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:59.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:23:59.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:23:59.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:23:59.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:59.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:23:59.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:59.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:23:59.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:59.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:23:59.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:23:59.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:00.097 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.095+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.098 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.095+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd8f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.098 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.095+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.098 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.095+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8ae3d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.099 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.095+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.099 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.095+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.099 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.099+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.099+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.099+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8ae3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.099+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c0d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.127+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.127+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.131 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.127+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.127+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.127+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.127+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928c480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd8f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.134 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.131+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8ae3f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.136 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.135+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.135+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.135+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.135+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.135+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8ae3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.137 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.135+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c0d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.145 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.143+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.145 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.143+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.179+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.179+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.183 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.183+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.183 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.183+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c1a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.186 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.183+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.186 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.183+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.186 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.183+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.187 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:24:00.183+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928c480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:00.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:00.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:00.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:00.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:00.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:01.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:01.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:01.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:01.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:01.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:01.079+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:01.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:01.083+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:01.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:01.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:01.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:01.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:01.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:01.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:01.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:01.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:01.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:01.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:01.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:02.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:02.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:02.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:02.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:02.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:24:02.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:02.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:02.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:02.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:02.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:02.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:02.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:02.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:02.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:02.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:02.747 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:24:02.747 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:02.750 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:02.761 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:04.144 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2336971439 2023-10-14T00:24:04.144 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:24:04.146 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243044 2023-10-14T00:24:04.146 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:04.146 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2336971439 ]] 2023-10-14T00:24:04.146 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2336971439 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\3\6\9\7\1\4\3\9 ]] 2023-10-14T00:24:04.147 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:24:04.447 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:04.443+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:04.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:04.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:04.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:04.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:04.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:04.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:07.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:07.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:07.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:07.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:07.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:07.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:07.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:07.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:07.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:07.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:07.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:07.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:07.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:07.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:07.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:07.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:07.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:07.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:07.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:07.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:07.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:07.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:08.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:08.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:08.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:08.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:08.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:08.127+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:08.143 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:08.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:08.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:08.147+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:08.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:08.147+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:08.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:08.147+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:08.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:08.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:08.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:08.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:08.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:08.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:08.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:08.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:08.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:08.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:08.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:11.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:11.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:11.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:11.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:11.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:11.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:11.197 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:11.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:11.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:11.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:11.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:11.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:11.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:11.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:12.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:12.283+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:12.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:12.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:12.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:12.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:12.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:12.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:12.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:12.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:12.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:12.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:13.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:13.355+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:13.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:13.355+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:13.366 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:13.363+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:13.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:13.379+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:13.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:13.379+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:13.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:13.379+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:13.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:13.379+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:14.150 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:14.151 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:24:14.154 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:14.170 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:14.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:14.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:14.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:14.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:14.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:14.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:14.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:14.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:14.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:14.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:14.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:14.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:14.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:14.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:15.431 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2336971439 2023-10-14T00:24:15.447 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:24:15.448 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243055 2023-10-14T00:24:15.449 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:15.449 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2336971439 ]] 2023-10-14T00:24:15.449 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2336971439 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\3\6\9\7\1\4\3\9 ]] 2023-10-14T00:24:15.449 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:24:15.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:15.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:15.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:15.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:15.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:15.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:15.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:15.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:15.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:15.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:16.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:16.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:16.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:16.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:16.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:16.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:16.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:16.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:16.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:16.931+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:16.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:16.931+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:16.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:16.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:16.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:16.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:16.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:16.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:16.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:16.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:16.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:16.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:16.975 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:16.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:17.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:17.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:17.501 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:17.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:17.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:17.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:19.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:19.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:19.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:19.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:19.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:19.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:19.397 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:19.395+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:19.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:19.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:19.449 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:19.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:19.449 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:19.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:20.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:20.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:20.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:20.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:20.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:20.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:20.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:20.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:20.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:20.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:21.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:21.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:21.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:21.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:21.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:21.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:21.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:21.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:21.392 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:21.603 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:21.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:21.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:21.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:21.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:21.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:21.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:21.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:21.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:21.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:21.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:21.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:21.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:24.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:24.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:24.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:24.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:24.732 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.731+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:24.732 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.731+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:24.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:24.763 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:24.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:24.776 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:24.776 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:24.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:25.467 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:24:25.468 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:25.468 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:25.468 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:25.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:25.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:25.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:25.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:25.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:25.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:25.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:25.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:25.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:25.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:25.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:26.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:25.999+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:26.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:26.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:26.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:26.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:26.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:26.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:26.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:26.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:26.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:26.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:26.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:26.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:26.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:26.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:26.972 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2336971439 2023-10-14T00:24:26.978 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:24:26.980 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243066 2023-10-14T00:24:26.980 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:26.980 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2336971439 ]] 2023-10-14T00:24:26.980 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2336971439 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\3\6\9\7\1\4\3\9 ]] 2023-10-14T00:24:26.980 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:24:27.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:27.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:27.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:27.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:27.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:27.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:27.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:27.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:27.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:27.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:27.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:27.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:27.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:27.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:27.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:27.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:28.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:28.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:28.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:28.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:28.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:28.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:28.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:28.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:28.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:28.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:28.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:28.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:28.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:28.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:28.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:28.555+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:28.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:28.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:28.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:28.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:28.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:28.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:28.777 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:24:28.775+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:24:29.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:29.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:29.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:29.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:29.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:29.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:29.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:29.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:29.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:29.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:29.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:29.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:29.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:29.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:30.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:30.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:30.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:30.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:30.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:30.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:30.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:30.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:30.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:30.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:31.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:31.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:31.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:31.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:31.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:31.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:31.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:31.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:31.895 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:31.895+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:31.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:31.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:31.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:31.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:31.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:31.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:31.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:31.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:31.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:31.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:31.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:31.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:34.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:34.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:34.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:34.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:34.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:34.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:34.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:34.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:34.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:34.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:35.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:35.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:35.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:35.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:35.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:35.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:35.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:35.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:36.997 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:36.999 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:24:37.000 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:37.000 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:38.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2336971439 2023-10-14T00:24:38.353 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:24:38.354 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243078 2023-10-14T00:24:38.354 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:38.355 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2336971439 ]] 2023-10-14T00:24:38.355 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2336971439 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\3\6\9\7\1\4\3\9 ]] 2023-10-14T00:24:38.355 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:24:39.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:39.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:39.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:39.122+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:39.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:39.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:39.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:39.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:39.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:39.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:39.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:39.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:40.936 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.934+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:24:40.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:40.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:40.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:40.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.970+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:40.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:40.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:40.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:40.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:41.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:41.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:41.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:41.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:41.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:41.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:41.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:41.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:41.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:41.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:41.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:41.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:41.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:41.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:42.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:42.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:42.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:42.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:42.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:42.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:42.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:42.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:42.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:42.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:42.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:42.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:42.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:42.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:42.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:42.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:42.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:42.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:42.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:42.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:42.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:42.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:42.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:42.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:42.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:42.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:42.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:42.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:43.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.154+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:43.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.154+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:43.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.162+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:43.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:43.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:43.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:43.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:43.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:43.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:43.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:43.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:43.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:43.864 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:43.869 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:43.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:44.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:24:44.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:24:44.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:44.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:44.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:44.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:44.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:44.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:44.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:44.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:44.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:44.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:44.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:44.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:44.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:44.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:44.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:24:44.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:44.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:44.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:44.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:24:44.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:44.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:44.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:44.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:47.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:47.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:47.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:47.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:47.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:47.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:48.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:48.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:48.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:48.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:48.103 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:48.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:48.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:48.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:48.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:48.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:48.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:48.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:48.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:48.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:48.375 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:24:48.376 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:48.381 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:48.387 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:48.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:48.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:48.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:48.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:48.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:48.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:49.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2336971439 2023-10-14T00:24:49.754 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:24:49.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243089 2023-10-14T00:24:49.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:49.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2336971439 ]] 2023-10-14T00:24:49.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2336971439 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\3\6\9\7\1\4\3\9 ]] 2023-10-14T00:24:49.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:24:52.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:52.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:52.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:52.309 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:52.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:52.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:52.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:52.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:52.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:52.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:52.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:52.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:52.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:52.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:52.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:52.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:52.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:52.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:52.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:52.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:52.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:52.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:52.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:53.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:24:53.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:53.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:53.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:53.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:53.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:53.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:53.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:53.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:54.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:54.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:54.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:54.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:54.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:24:54.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:54.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:54.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:54.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:54.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:54.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:55.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:55.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:55.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:55.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:57.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:57.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:57.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:57.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:57.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:57.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:57.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:57.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:57.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:57.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:57.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:57.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:57.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:57.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:24:57.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:57.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:57.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:57.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:24:57.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:57.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:57.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:57.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:24:57.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:57.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:24:57.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:57.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:24:58.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:58.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:24:58.183 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:24:58.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:24:59.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:24:59.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:24:59.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:59.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:59.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:59.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:24:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:24:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:24:59.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:24:59.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:24:59.758 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:59.759 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:24:59.760 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:59.762 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:00.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:00.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:00.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:00.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:00.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:00.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:00.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:00.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:00.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:00.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:00.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:00.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:00.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:00.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:01.134 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2561721673 2023-10-14T00:25:01.135 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:25:01.136 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243101 2023-10-14T00:25:01.136 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2561721673 ]] 2023-10-14T00:25:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2561721673 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\3\3\6\9\7\1\4\3\9 ]] 2023-10-14T00:25:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2561721673 2023-10-14T00:25:02.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:02.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:02.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:02.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:02.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:02.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:02.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:02.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:02.600 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:25:02.598+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve create_timestamp: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:02.600 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:25:02.598+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:03.550 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2561721673 until 2023-10-14T01:25:02.867201+0000 (3600 sec) 2023-10-14T00:25:03.657 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2561721673 2023-10-14T00:25:03.671 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:25:03.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:03.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:03.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:03.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:03.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:03.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:04.937 INFO:tasks.workunit.client.0.smithi138.stderr:listed 27 entries 2023-10-14T00:25:05.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2561721673 2023-10-14T00:25:05.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:25:05.695 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:05.694+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:05.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:05.702+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:05.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:05.702+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:05.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:05.702+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:05.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:05.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:05.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:05.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:05.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:05.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:05.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:05.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:05.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:05.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:07.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:07.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:07.197 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:07.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:07.198 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:07.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:07.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:07.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:07.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:07.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:07.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:07.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:08.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:08.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:08.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:08.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:08.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:08.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:08.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:08.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:08.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:08.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:08.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:08.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:08.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:08.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:09.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:09.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:09.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:09.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:09.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:09.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:09.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:09.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:25:09.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:25:09.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:09.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:09.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:09.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:09.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:09.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:09.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:09.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:09.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:10.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:10.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:10.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:10.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:10.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:10.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:10.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:10.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:10.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:10.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:10.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:11.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:11.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:11.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:11.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:11.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:11.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:11.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:11.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:11.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:11.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:11.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:11.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:11.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:11.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:13.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:13.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:13.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:13.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:13.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:13.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:13.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:13.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:13.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:13.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:13.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:13.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:14.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:14.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:14.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:14.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:14.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:14.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:15.010 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:15.010 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:15.010 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:15.016 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:25:15.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:15.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:15.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:15.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:15.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:15.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:15.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:15.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:15.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:15.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:16.271 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1887320117 2023-10-14T00:25:16.275 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:25:16.282 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243116 2023-10-14T00:25:16.282 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:16.283 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1887320117 ]] 2023-10-14T00:25:16.283 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1887320117 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\6\1\7\2\1\6\7\3 ]] 2023-10-14T00:25:16.283 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/1887320117 2023-10-14T00:25:16.645 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:16.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:16.645 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:16.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:16.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:16.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:16.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:16.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:16.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:16.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:16.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:16.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:16.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:16.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:16.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:16.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:16.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:16.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:16.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:16.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:16.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:16.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:16.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:16.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:16.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:16.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:18.369 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/1887320117 until 2023-10-14T01:25:17.798106+0000 (3600 sec) 2023-10-14T00:25:18.410 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:25:18.410 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/1887320117 2023-10-14T00:25:19.471 INFO:tasks.workunit.client.0.smithi138.stderr:listed 28 entries 2023-10-14T00:25:19.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:19.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:19.487 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:19.486+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:19.553 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/1887320117 2023-10-14T00:25:19.553 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:25:19.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:19.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:19.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:19.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:19.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:19.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:19.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:19.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:19.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:19.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:19.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:19.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:19.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:19.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:20.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:20.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:20.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:20.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:20.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:20.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:21.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:21.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:21.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:21.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:21.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:21.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:21.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:21.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:21.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:21.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:21.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:21.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:21.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:21.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:24.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:24.430+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:24.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:24.430+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:24.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:24.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:24.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:24.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:24.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:24.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:24.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:24.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:24.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:24.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:25.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:25.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:25.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:25.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:25.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:25.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:25.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:25.186+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:25.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:25.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:25.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:25.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:25.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:25.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:26.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:26.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:26.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:26.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:26.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:26.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:26.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:26.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:26.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:27.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:27.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:27.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:27.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:27.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:27.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:27.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:27.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:27.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:27.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:28.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:28.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:28.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:28.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:28.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:28.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:28.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:28.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:28.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:29.560 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:29.563 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:29.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:29.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:29.566 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:29.567 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:25:29.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:29.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:29.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:29.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:29.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:29.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:29.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:29.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:30.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:30.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:30.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:30.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:30.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:30.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:30.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:30.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:30.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:30.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:30.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:30.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:30.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:30.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:30.929 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1887320117 2023-10-14T00:25:30.930 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:25:30.931 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243130 2023-10-14T00:25:30.931 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:30.931 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1887320117 ]] 2023-10-14T00:25:30.931 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1887320117 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\8\7\3\2\0\1\1\7 ]] 2023-10-14T00:25:30.932 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:25:31.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:31.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:31.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:31.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:31.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:31.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:31.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:31.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:32.580 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:32.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:32.580 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:32.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:32.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:32.593+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:32.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:32.609+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:32.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:32.617+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:32.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:32.617+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:32.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:32.621+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:34.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:34.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:34.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:34.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:34.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:34.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:34.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:34.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:34.432 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:34.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:34.432 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:34.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:34.432 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:34.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:34.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:34.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:34.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:34.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:34.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:35.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:35.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:35.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:35.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:35.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:35.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:35.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:35.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:35.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:35.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:35.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:35.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:36.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:36.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:36.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:36.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:36.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:36.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:36.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:36.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:36.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:36.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:36.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:36.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:37.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:37.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:37.410 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:37.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:37.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:37.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:37.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:37.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:37.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:37.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:37.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:37.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:37.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:38.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:38.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:38.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:38.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:38.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:38.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:38.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:38.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:38.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:38.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:38.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:38.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:38.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:38.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:38.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:38.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:38.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:38.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:25:38.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:38.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:38.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:38.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:38.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:38.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:38.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:38.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:38.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:38.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:38.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:39.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:39.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:38.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:39.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:39.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:40.933 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:25:40.934 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:40.934 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:40.948 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:41.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:41.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:41.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:41.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:41.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:41.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:41.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:41.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:41.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:41.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:41.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:42.603 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1887320117 2023-10-14T00:25:42.603 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:25:42.604 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243142 2023-10-14T00:25:42.604 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:42.604 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1887320117 ]] 2023-10-14T00:25:42.605 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1887320117 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\8\7\3\2\0\1\1\7 ]] 2023-10-14T00:25:42.605 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:25:42.726 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:42.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:42.791 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:42.789+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:43.326 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:43.321+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:43.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:43.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:44.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:44.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:44.008 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:44.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:44.008 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:44.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:44.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:44.013+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:44.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:44.013+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:44.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:44.013+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:44.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:44.013+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:46.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:46.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:46.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:46.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:46.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:46.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:46.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:46.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:46.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:48.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:48.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:48.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:48.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:48.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:25:48.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:48.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:48.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:48.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:49.219 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:49.213+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:49.226 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:49.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:49.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:49.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:49.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:49.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:49.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:49.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:49.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:49.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:50.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:50.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:50.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:50.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:50.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:50.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:50.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:50.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:50.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:50.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:50.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:50.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:50.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:50.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:51.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:51.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:25:51.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:51.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:25:51.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:51.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:25:52.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:52.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:52.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:52.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:52.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:52.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:52.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:52.614 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:25:52.621 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:52.621 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:52.632 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:52.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:52.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:52.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:52.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:52.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:52.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:52.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:52.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:52.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:53.991 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1887320117 2023-10-14T00:25:53.991 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:25:53.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243153 2023-10-14T00:25:53.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:53.993 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1887320117 ]] 2023-10-14T00:25:53.993 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1887320117 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\8\7\3\2\0\1\1\7 ]] 2023-10-14T00:25:53.993 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:25:54.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:54.361+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:54.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:54.429+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:54.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:54.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:54.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:54.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:54.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:54.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:54.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:54.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:25:54.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:54.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:54.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:54.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:25:54.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:54.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:54.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:54.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:54.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:54.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:54.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:54.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:54.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:54.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:55.645 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:55.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:55.645 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:55.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:56.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:56.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:57.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:57.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:25:57.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:57.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:57.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:57.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:57.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:57.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:57.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:57.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:57.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:57.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:59.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:59.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:59.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:59.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:59.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:25:59.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:25:59.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:25:59.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:25:59.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:25:59.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:25:59.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:25:59.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.709+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:25:59.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.709+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:25:59.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.761+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:25:59.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.769+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:25:59.785 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:25:59.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:25:59.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:25:59.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:00.027 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.029+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.027 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.029+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8bfc080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.029+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.029+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.029+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.029+0000 7f082630d640 -1 librbd::ImageState: 0x5627d89c1a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.035 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.037+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.035 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.037+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9108180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.061+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.061+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.061+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.061+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.061+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.061+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dea280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.073+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.073+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.077+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.077+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.099 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.101+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.099 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.101+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.101+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.101+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.101+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928c880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.101+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae2000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.109+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.109+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.109+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.109+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928ce80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.109+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.109+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.117+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.117+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.117 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.117+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.118 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.117+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928d480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.155 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.153+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.155 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.153+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.155 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.157+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928d880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.156 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.157+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.156 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.157+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.156 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:26:00.157+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928d680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:00.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:00.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:00.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:00.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:00.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:00.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:00.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:00.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:00.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:00.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:00.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:00.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:00.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:01.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:26:01.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:26:01.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:01.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:01.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:01.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:01.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:01.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:01.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:01.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:01.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:01.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:02.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:02.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:02.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:02.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:02.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:02.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:02.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:02.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:02.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:02.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:02.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:02.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:03.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:03.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:03.994 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:26:03.994 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:03.996 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:04.005 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:04.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:04.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:04.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:04.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:04.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:04.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:04.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:04.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:04.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:04.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:04.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:04.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:04.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:04.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1887320117 2023-10-14T00:26:05.560 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:26:05.560 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243165 2023-10-14T00:26:05.560 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:05.560 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1887320117 ]] 2023-10-14T00:26:05.561 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1887320117 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\8\7\3\2\0\1\1\7 ]] 2023-10-14T00:26:05.561 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:26:08.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:08.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:08.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:08.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:08.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:08.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.825+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:08.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:08.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:08.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:08.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:08.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:08.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:08.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:08.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:08.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:08.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:09.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:08.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:09.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:08.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:09.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:08.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:10.992 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:10.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:11.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:11.021+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:11.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:11.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:11.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:11.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:11.048 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:11.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:12.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:12.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:12.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:12.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:12.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:12.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:12.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:12.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:12.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:12.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:13.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:13.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:13.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:13.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:13.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:13.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:13.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:13.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:13.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:26:13.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:13.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:13.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:13.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:13.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:13.992 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:13.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:13.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:13.997+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:14.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:14.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:14.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:14.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:14.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:14.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:14.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:14.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:14.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:14.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:14.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:14.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:14.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:14.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:14.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:14.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:15.562 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:26:15.562 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:15.563 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:15.591 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:16.445 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:16.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:26:16.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:16.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:16.460 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:16.461+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:16.934 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1887320117 2023-10-14T00:26:16.935 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:26:16.936 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243176 2023-10-14T00:26:16.937 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:16.937 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1887320117 ]] 2023-10-14T00:26:16.937 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1887320117 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\8\7\3\2\0\1\1\7 ]] 2023-10-14T00:26:16.937 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:26:18.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:18.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:18.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:18.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:18.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:18.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:18.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:18.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:18.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:18.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:18.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:18.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:18.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:18.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:18.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:18.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:18.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:18.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:18.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:18.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:18.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:18.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:18.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:19.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:19.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:19.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:19.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:19.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:19.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:19.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:19.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:19.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:19.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:19.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:19.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:19.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:19.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:20.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:20.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:20.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:20.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:20.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:20.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:20.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:20.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:20.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:20.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:20.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:20.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:21.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:21.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:21.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:21.021+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:21.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:21.021+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:21.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:21.021+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:22.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:22.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:26:22.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:22.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:22.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:22.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:22.256 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:22.256+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:22.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:22.264+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:22.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:22.264+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:22.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:22.264+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:22.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:22.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:22.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:22.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:22.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:22.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:22.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:22.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:22.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:22.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:22.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:22.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:23.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:23.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:23.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:23.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:23.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:23.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:23.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:23.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:23.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:23.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:23.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:23.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:23.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:23.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:23.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:23.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:23.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:23.643 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:23.644+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:23.667 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:23.668+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:23.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:23.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:23.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:23.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:23.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:23.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:23.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:24.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:24.252+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:26:24.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:24.264+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:24.268 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:24.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:24.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:24.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:24.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:24.328+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:24.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:24.328+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:24.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:24.328+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:24.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:24.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:24.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:24.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:24.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:24.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:24.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:26:24.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:26:24.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:24.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:24.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:24.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:24.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:24.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:24.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:24.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:24.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:24.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:24.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:24.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:24.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:25.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:25.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:25.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:25.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:25.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:25.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:25.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:25.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:26.949 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:26:26.949 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:26.950 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:26.959 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:28.308 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1611482965 2023-10-14T00:26:28.323 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:26:28.326 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243188 2023-10-14T00:26:28.326 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:28.326 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1611482965 ]] 2023-10-14T00:26:28.326 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1611482965 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\8\8\7\3\2\0\1\1\7 ]] 2023-10-14T00:26:28.327 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/1611482965 2023-10-14T00:26:28.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:28.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:28.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:28.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:28.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:28.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:28.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:28.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:29.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:29.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:29.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:29.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:29.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:26:29.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:29.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:29.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:29.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:29.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:29.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:29.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:29.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:29.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:26:29.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:29.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:29.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:29.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:29.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:29.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:29.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:30.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:30.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:30.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:30.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:30.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:30.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:30.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:30.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:30.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:30.387 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/1611482965 until 2023-10-14T01:26:29.738471+0000 (3600 sec) 2023-10-14T00:26:30.422 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:26:30.427 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/1611482965 2023-10-14T00:26:31.744 INFO:tasks.workunit.client.0.smithi138.stderr:listed 29 entries 2023-10-14T00:26:31.805 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/1611482965 2023-10-14T00:26:31.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:26:32.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:32.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:32.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:32.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:32.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:32.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:32.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:32.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:34.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:34.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:34.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:34.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:34.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:34.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:36.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:36.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:36.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:36.800+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:36.800 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:36.800+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:36.800 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:36.800+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:37.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:37.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:37.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:37.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:37.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:37.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:37.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:37.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:37.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:37.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:37.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:37.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:39.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.552+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:39.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.556+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:39.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:39.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:39.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:39.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:39.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:26:39.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:39.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:40.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:40.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:40.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:40.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:40.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:40.141 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.140+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:26:40.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.140+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:40.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:40.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:40.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:40.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:40.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:40.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:41.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:41.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:41.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:41.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:41.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:41.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:41.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:41.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:41.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:41.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:41.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:41.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:41.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:41.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:41.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:41.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:41.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:41.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:41.809 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:26:41.810 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:41.810 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:41.814 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:42.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:42.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:42.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:42.396 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:42.396 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:42.397 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:42.789 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.788+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:42.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.788+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:42.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:42.842 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:42.843 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:42.843 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:42.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:42.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:42.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:42.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:42.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:42.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:42.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:43.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:43.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:43.154 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1611482965 2023-10-14T00:26:43.154 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:26:43.154 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243203 2023-10-14T00:26:43.155 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:43.155 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1611482965 ]] 2023-10-14T00:26:43.155 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1611482965 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\6\1\1\4\8\2\9\6\5 ]] 2023-10-14T00:26:43.155 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:26:44.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:44.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:44.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:44.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:44.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:44.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:44.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:44.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:44.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:44.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:45.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:45.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:45.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:45.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:45.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:45.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:45.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:45.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:46.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:46.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:46.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:46.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:46.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:46.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:46.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:46.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:46.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:46.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:46.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:46.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:47.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:47.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:26:47.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:47.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:26:47.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:47.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:47.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:47.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:47.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:47.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:47.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:47.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:47.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:47.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:26:47.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:47.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:47.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:47.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:47.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:47.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:47.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:47.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:47.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:47.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:47.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:47.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:47.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:47.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:48.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:26:48.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:26:48.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:26:48.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:48.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:26:48.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:48.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:48.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:48.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:48.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:48.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:49.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:49.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:49.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:49.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:49.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:49.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:50.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:50.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:50.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:50.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:50.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:50.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:50.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:50.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:50.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:50.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:51.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:51.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:26:51.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:51.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:51.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:51.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:51.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:51.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:53.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:53.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:53.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:53.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:53.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:53.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:53.182 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:53.183 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:26:53.192 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:53.205 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:54.657 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1611482965 2023-10-14T00:26:54.658 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:26:54.660 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243214 2023-10-14T00:26:54.660 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:54.660 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1611482965 ]] 2023-10-14T00:26:54.660 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1611482965 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\6\1\1\4\8\2\9\6\5 ]] 2023-10-14T00:26:54.660 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:26:55.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:55.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:55.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:55.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:55.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:55.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:56.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:56.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:26:56.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:56.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:56.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:56.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:26:56.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:56.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:58.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:58.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:26:58.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:58.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:26:58.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:26:58.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:26:59.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:59.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:26:59.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:59.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:26:59.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:59.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:26:59.463 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:59.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:26:59.463 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:26:59.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.004+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.004+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8860a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.005 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.004+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.005 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.004+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.009 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.008+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.009 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.008+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd8c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.028+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.028+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd6480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.032+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.032+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d89c1000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.058 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.056+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.059 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.056+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.059 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.056+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9109380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.059 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.056+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d89c1280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.108+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.108+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8deba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.108+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.108+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.112+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.114 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.112+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae2080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.175 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.172+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.176 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.172+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.176 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.172+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae3600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.176 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.172+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d7bfa080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.176 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.176+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.176 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.176+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d905e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.190 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.188+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.191 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.188+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d905fe00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.191 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.192 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.192 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.192 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.192 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.192 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.193 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae3f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.193 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d905d280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.193 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d81a8a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.193 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8879680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.193 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d928df00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.194 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:00.192+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d928c280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:27:00.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:00.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:00.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:00.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:00.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:00.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:04.664 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:04.667 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:04.674 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:27:04.675 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:05.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:05.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:05.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:05.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:05.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:05.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:05.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:05.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:05.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:06.195 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1611482965 2023-10-14T00:27:06.199 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:27:06.201 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243226 2023-10-14T00:27:06.201 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:06.202 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1611482965 ]] 2023-10-14T00:27:06.202 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1611482965 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\6\1\1\4\8\2\9\6\5 ]] 2023-10-14T00:27:06.202 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:27:06.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:06.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:06.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:06.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:06.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:06.968+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:06.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:06.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:06.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:06.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:06.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:06.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:12.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:12.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:12.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:12.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:12.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:12.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:12.406 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:12.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:12.406 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:12.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:13.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:13.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:13.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:13.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:13.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:13.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:13.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:13.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:13.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:13.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:13.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:13.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:15.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:15.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:15.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:15.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:15.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:15.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:15.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:15.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:15.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:15.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:15.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:15.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:16.207 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:27:16.207 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:16.207 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:16.220 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:17.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:17.192+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:27:17.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:17.192+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:17.207 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:17.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:17.233 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:17.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:17.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:17.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:17.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:17.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:17.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:17.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:17.465 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1611482965 2023-10-14T00:27:17.471 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:27:17.472 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243237 2023-10-14T00:27:17.472 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:17.473 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1611482965 ]] 2023-10-14T00:27:17.473 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1611482965 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\6\1\1\4\8\2\9\6\5 ]] 2023-10-14T00:27:17.473 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:27:18.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:18.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:27:18.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:18.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:18.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:18.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:18.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:18.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:18.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:18.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:18.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:18.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:18.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:18.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:19.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:19.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:19.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:19.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:19.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:19.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:20.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:20.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:20.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:20.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:20.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:20.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:20.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:20.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:20.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:20.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:20.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:20.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:20.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:20.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:21.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:21.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:21.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:21.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:21.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:21.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:21.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:21.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:21.619 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:21.615+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:21.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:21.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:21.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:21.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:21.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:21.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:21.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:21.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:22.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:22.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:22.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:22.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:22.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:22.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:22.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:22.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:22.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:23.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:23.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:23.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:23.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:23.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:23.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:23.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:23.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:23.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:23.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:23.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:23.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:24.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:24.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:24.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:24.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:24.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:24.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:25.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:25.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:25.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:25.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:25.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:25.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:25.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:25.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:25.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:25.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:25.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:25.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:25.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:26.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:26.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:26.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:26.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:26.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:26.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:26.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:26.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:27:26.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:26.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:26.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:26.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:26.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:26.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:26.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:27:26.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:26.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:26.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:26.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:26.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:26.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:26.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:27:26.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:26.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:26.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:26.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:26.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:26.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:26.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:26.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:26.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:27.483 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:27.483 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:27:27.484 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:27.490 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:27.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:27.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:27.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:27.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:27.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:27.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:27.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:27.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:27.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:28.583 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1611482965 2023-10-14T00:27:28.584 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:27:28.584 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243248 2023-10-14T00:27:28.585 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:28.585 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1611482965 ]] 2023-10-14T00:27:28.585 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1611482965 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\6\1\1\4\8\2\9\6\5 ]] 2023-10-14T00:27:28.585 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:27:28.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:28.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:28.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:28.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:29.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:29.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:29.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:29.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:29.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:29.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:29.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:29.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:29.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:29.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:29.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:29.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:29.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:29.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:29.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:29.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:29.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:29.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:32.235 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:32.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:32.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:32.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:32.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:32.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:32.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:32.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:32.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:32.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:32.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:32.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:32.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:32.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:35.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:27:35.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:35.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:35.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:35.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:35.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:35.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:35.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:35.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:35.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:35.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:35.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:35.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:35.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:35.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:35.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:36.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:36.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:36.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:36.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:36.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:36.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:36.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:36.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:36.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:36.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:38.590 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:38.593 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:27:38.594 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:38.595 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:40.196 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2983772729 2023-10-14T00:27:40.197 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:27:40.198 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243260 2023-10-14T00:27:40.198 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:40.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2983772729 ]] 2023-10-14T00:27:40.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2983772729 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\6\1\1\4\8\2\9\6\5 ]] 2023-10-14T00:27:40.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2983772729 2023-10-14T00:27:40.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:40.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:40.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:40.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:40.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:40.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:40.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:40.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:40.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:40.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:40.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:40.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:42.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:42.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:42.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:42.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:42.409 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2983772729 until 2023-10-14T01:27:41.757059+0000 (3600 sec) 2023-10-14T00:27:42.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:42.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:42.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:42.476 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:27:42.497 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2983772729 2023-10-14T00:27:42.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:42.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:43.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:43.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:43.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:43.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:43.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:43.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:43.691 INFO:tasks.workunit.client.0.smithi138.stderr:listed 30 entries 2023-10-14T00:27:43.720 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2983772729 2023-10-14T00:27:43.720 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:27:44.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:44.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:44.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:44.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:44.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:44.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:44.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:44.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:44.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:44.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:45.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:45.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:45.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:45.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:45.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:46.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:46.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:46.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:46.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:46.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:46.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:46.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:46.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:46.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:46.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:46.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:46.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:47.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:47.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:27:47.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:47.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:47.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:47.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:47.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:47.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:47.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:47.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:47.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:47.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:47.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:47.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:48.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:48.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:48.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:48.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:48.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:48.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:48.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:48.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:48.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:48.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:48.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:48.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:48.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:48.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:48.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:49.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:49.795+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:49.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:49.807+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:49.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:49.811+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:49.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:49.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:49.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:49.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:49.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:49.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:50.229 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:50.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:50.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:50.235+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:50.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:50.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:50.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:50.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:50.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:50.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:53.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:53.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:53.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:53.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:53.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:53.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:53.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:53.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:53.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:53.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:53.722 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:27:53.723 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:53.730 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:53.734 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:54.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:54.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:54.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:54.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:54.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:54.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:54.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:54.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:54.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:54.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:55.126 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:55.174 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2983772729 2023-10-14T00:27:55.174 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:27:55.176 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243275 2023-10-14T00:27:55.176 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:55.176 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2983772729 ]] 2023-10-14T00:27:55.176 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2983772729 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\8\3\7\7\2\7\2\9 ]] 2023-10-14T00:27:55.176 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:27:55.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:55.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:55.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:55.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:55.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:55.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:55.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:55.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:55.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:56.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:56.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:56.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:56.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:56.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:56.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:56.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:56.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:56.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:56.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:56.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:56.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:57.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:57.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:57.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:57.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:57.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:57.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:57.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:57.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:57.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:57.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:57.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:57.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:57.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:58.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:27:58.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:27:58.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:27:58.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:58.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:58.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:58.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:58.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:58.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:27:58.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:58.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:27:58.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:58.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:27:58.569 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:58.567+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:27:58.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:58.599+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:27:58.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:58.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:27:58.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:58.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:27:58.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:27:58.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:27:59.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:59.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:27:59.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:59.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:27:59.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:59.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:27:59.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:59.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:27:59.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:27:59.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:00.002 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:59.999+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.003 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:27:59.999+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.021 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.019+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.021 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.019+0000 7f082630d640 -1 librbd::ImageState: 0x5627d896de00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.022 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.019+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.022 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.019+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.019+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.019+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8b60e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.035+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.035+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd7b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.043+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.043+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.051+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.051+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.059+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.059+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.075+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.075+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.082 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.079+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.082 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.079+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928d800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.083+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.083+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.090 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.087+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.090 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.087+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e16b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.103+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.103+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928c780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.103+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.103+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928cf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.120 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.119+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.120 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.119+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928c800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.121 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.119+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.121 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.119+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.131+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.131+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.159+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.159+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f84000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.163 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.163+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.163 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.163+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f84200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.178 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.175+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.179 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:28:00.175+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f84400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:28:00.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:00.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:00.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:00.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:00.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:00.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:00.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:00.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:00.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:00.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:00.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:00.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:00.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:00.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:02.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:02.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:02.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:02.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:02.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:02.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:02.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:02.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:03.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:03.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:04.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:04.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:04.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:04.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:04.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:04.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:04.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:04.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:05.183 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:05.184 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:05.186 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:05.188 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:28:05.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:05.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:06.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:06.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:06.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:06.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:06.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:06.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:06.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:06.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:06.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:06.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:06.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:06.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:06.508 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2983772729 2023-10-14T00:28:06.509 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:28:06.509 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243286 2023-10-14T00:28:06.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:06.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2983772729 ]] 2023-10-14T00:28:06.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2983772729 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\8\3\7\7\2\7\2\9 ]] 2023-10-14T00:28:06.510 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:28:07.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:07.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:07.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:07.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:07.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:07.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:07.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:07.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:07.466 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:07.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:07.467 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:07.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:07.467 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:07.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:08.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:08.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:08.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:08.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:08.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:08.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:08.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:08.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:08.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:08.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:08.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:08.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:08.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:08.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:09.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:09.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:09.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:09.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:09.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:09.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:09.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:09.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:09.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:09.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:10.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:28:10.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:28:10.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:10.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:10.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:10.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:10.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:10.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:10.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:10.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:10.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:10.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:10.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:10.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:10.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:10.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:10.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:11.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:11.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:11.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:11.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:11.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:11.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:11.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:11.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:11.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:11.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:11.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:11.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:12.121 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:12.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:12.126 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:12.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:12.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:12.135+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:13.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:13.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:13.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:13.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:13.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:13.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:13.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:13.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:14.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:14.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:14.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:14.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:14.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:14.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:14.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:14.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:14.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:14.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:15.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:15.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:15.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:15.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:15.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:15.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:15.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:15.735+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:15.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:15.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:15.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:15.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:15.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:15.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:15.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:15.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:15.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:15.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:15.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:15.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:15.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:15.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:15.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:16.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:16.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:16.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:16.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:16.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:16.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:16.519 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:16.520 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:16.521 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:16.522 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:28:17.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:17.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:17.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:17.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:17.194 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:17.191+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:17.194 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:17.191+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:17.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:17.191+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:17.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:17.191+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:17.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:17.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:17.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:17.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:17.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:17.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:17.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:17.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:17.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:17.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:17.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:17.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:17.934 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2983772729 2023-10-14T00:28:17.941 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:28:17.942 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243297 2023-10-14T00:28:17.942 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:17.942 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2983772729 ]] 2023-10-14T00:28:17.943 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2983772729 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\8\3\7\7\2\7\2\9 ]] 2023-10-14T00:28:17.943 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:28:18.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:18.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:18.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:18.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:18.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:18.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:18.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:18.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:18.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:18.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:19.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:19.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:20.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:20.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:20.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:20.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:20.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:20.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:20.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:20.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:20.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:20.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:28:20.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:28:20.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:20.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:20.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:20.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:21.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:21.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:21.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:21.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:21.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:21.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:21.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:21.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:21.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:21.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:23.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:23.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:23.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:23.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:23.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:23.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:23.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:23.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:23.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:23.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:23.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:23.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:23.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:23.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:24.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:28:24.230 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:24.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:24.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:24.239 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:24.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:24.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:24.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:24.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:24.732 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:24.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:24.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:24.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:24.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:24.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:25.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:28:25.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:28:25.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:25.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:25.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:25.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:25.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:25.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:25.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:25.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:25.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:26.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:26.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:26.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:26.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:26.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:26.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:26.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:26.514+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:27.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:27.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:27.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:27.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:27.952 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:27.954 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:27.954 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:28:29.406 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2983772729 2023-10-14T00:28:29.414 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:28:29.415 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243309 2023-10-14T00:28:29.416 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:29.416 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2983772729 ]] 2023-10-14T00:28:29.416 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2983772729 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\8\3\7\7\2\7\2\9 ]] 2023-10-14T00:28:29.416 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:28:29.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:29.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:29.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:29.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:29.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:29.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:29.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:29.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:29.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:30.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:30.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:31.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:31.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:31.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:31.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:31.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:32.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:32.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:32.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:32.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:33.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:33.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:33.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:33.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:33.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:33.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:33.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:33.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:35.009 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:35.009 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:35.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.014+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:35.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:35.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:35.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:35.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:35.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:35.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:35.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:35.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:35.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:35.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:35.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:35.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:28:35.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.970+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:35.971 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.970+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:35.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:35.978+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:36.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:36.014+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:36.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:36.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:36.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:36.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:36.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:36.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:36.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:36.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:36.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:36.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:36.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:36.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:36.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:36.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:36.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:36.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:36.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:36.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:36.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:36.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:36.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:36.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:36.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:37.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:37.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:37.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:37.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:37.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:37.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:37.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:37.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:37.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:37.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:37.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:37.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:37.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:37.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:38.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:38.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:38.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:38.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:38.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:38.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:38.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:38.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:38.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:38.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:38.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:39.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:39.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:38.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:39.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:39.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:39.424 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:28:39.424 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:39.425 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:39.435 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:39.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:39.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:39.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:39.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:39.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:39.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:39.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:39.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:40.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:40.246 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:40.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:40.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:40.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:40.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:40.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:40.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:40.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:40.290+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:40.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:28:40.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:28:40.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:40.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:40.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:40.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:40.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:40.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:40.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:40.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:40.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:40.855 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2983772729 2023-10-14T00:28:40.855 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:28:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243320 2023-10-14T00:28:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2983772729 ]] 2023-10-14T00:28:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2983772729 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\8\3\7\7\2\7\2\9 ]] 2023-10-14T00:28:40.857 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:28:41.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:41.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:41.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:41.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:41.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:41.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:41.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:41.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:41.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:41.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:41.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:41.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:41.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:41.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:41.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:41.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:42.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:42.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:42.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:42.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:42.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:42.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:42.107 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:42.106+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:42.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:42.106+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:42.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:42.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:42.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:42.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:42.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:42.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:42.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:42.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:42.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:42.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:42.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:42.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:42.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:42.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:43.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:43.886+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:43.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:43.886+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:43.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:43.922+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:43.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:43.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:43.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:43.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:43.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:43.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:43.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:43.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:44.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:44.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:44.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:44.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:44.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:44.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:44.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:44.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:44.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:44.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:44.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:45.844 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:45.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:28:45.844 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:45.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:45.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:45.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:45.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:45.854+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:45.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:45.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:45.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:45.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:45.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:45.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:46.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:46.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:28:46.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:46.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:46.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:46.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:46.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:46.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:46.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:46.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:46.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:46.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:46.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:46.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:46.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:46.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:46.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:47.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:47.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:47.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:47.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:47.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:47.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:47.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:47.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:47.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:47.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:47.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:47.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:47.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:47.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:48.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:48.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:48.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:48.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:48.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:48.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:48.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:48.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:48.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:48.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:48.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:48.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:48.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:49.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:49.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:49.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:49.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:49.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:49.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:49.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:49.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:49.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:50.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:50.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:50.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:50.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:50.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:50.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:50.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:50.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:50.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:50.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:50.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:50.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:50.863 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:50.867 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:50.871 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:50.873 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:28:52.150 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3862910917 2023-10-14T00:28:52.151 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:28:52.152 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243332 2023-10-14T00:28:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3862910917 ]] 2023-10-14T00:28:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3862910917 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\8\3\7\7\2\7\2\9 ]] 2023-10-14T00:28:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3862910917 2023-10-14T00:28:52.788 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:52.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:52.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:52.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:52.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:52.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:52.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:52.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:53.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:28:53.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:53.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:53.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:53.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:53.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:53.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:53.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:53.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:53.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:53.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:53.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:53.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:54.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:28:54.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:54.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:54.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:54.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:54.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:54.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:54.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:54.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:54.436 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3862910917 until 2023-10-14T01:28:53.820150+0000 (3600 sec) 2023-10-14T00:28:54.485 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:28:54.509 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3862910917 2023-10-14T00:28:54.898 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:54.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:28:54.928 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:54.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:54.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:54.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:54.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:54.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:54.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:55.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:55.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:55.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:55.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:55.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:55.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:55.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:55.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:55.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:55.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:55.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:55.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:55.857 INFO:tasks.workunit.client.0.smithi138.stderr:listed 31 entries 2023-10-14T00:28:55.890 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3862910917 2023-10-14T00:28:55.890 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:28:55.984 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:55.978+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:28:56.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:55.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:28:56.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:56.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:28:56.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:56.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:28:56.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:56.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:28:56.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:56.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:28:56.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:56.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:56.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:28:56.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:56.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:56.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:56.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:56.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:56.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:56.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:56.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:56.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:58.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:58.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:28:58.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:58.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:58.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:58.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:58.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:58.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:58.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:58.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:58.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:58.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:28:59.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:59.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:28:59.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:59.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:28:59.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:59.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:28:59.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:59.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:28:59.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:28:59.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:00.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:59.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:29:00.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:28:59.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:00.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:00.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:00.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:00.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.010+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.010+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd9880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.010+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.010+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae3000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.010+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.010+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.016 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.014+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.016 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.014+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d905f980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.026+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.026+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:00.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:00.046 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.046+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.046 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.046+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.050+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.050+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:00.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:00.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:00.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.054+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.054+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.060 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.058+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.060 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.058+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.066+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.070+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.070+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.088 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.086+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.088 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.086+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.098 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.094+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.098 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.094+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.104 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.102+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.102+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.102+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.102+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.106 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.106 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.109 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.126+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.126+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.126+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.126+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9358100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.126+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9358300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:29:00.126+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9358500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:29:01.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:01.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:01.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:01.486+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:01.504 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:01.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:01.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:01.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:01.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:01.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:01.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:01.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:01.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:01.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:01.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:01.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:01.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:01.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:01.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:01.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:01.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:01.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:02.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:02.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:29:02.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:02.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:02.859 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:02.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:02.859 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:02.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:02.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:02.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:03.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:03.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:03.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:03.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:03.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:03.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:03.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:03.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:03.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:03.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:03.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:03.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:03.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:03.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:04.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:04.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:04.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:04.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:04.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:04.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:04.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:04.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:04.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:04.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:04.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:04.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:04.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:04.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:05.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:05.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:05.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:05.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:05.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:05.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:05.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:05.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:05.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:05.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:05.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:05.898 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:05.918 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:29:05.919 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:05.925 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:06.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:06.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:06.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:06.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:06.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:06.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:06.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:06.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:06.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:06.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:06.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:06.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:07.317 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3862910917 2023-10-14T00:29:07.318 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:29:07.318 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243347 2023-10-14T00:29:07.318 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:07.319 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3862910917 ]] 2023-10-14T00:29:07.319 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3862910917 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\2\9\1\0\9\1\7 ]] 2023-10-14T00:29:07.319 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:29:08.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:08.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:08.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:08.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:08.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:08.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:08.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:08.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:08.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:08.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:08.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:08.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:08.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:08.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:10.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:10.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:10.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:10.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:10.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:10.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:10.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:10.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:14.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:14.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:14.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:14.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:14.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:14.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:14.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:14.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:16.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:16.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:16.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:16.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:16.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:16.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:16.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:16.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:16.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:16.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:16.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:17.322 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:17.323 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:29:17.328 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:17.342 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:17.548 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:17.545+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:17.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:17.585+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:17.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:17.585+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:17.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:17.585+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:18.731 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3862910917 2023-10-14T00:29:18.733 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:29:18.734 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243358 2023-10-14T00:29:18.734 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:18.735 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3862910917 ]] 2023-10-14T00:29:18.735 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3862910917 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\2\9\1\0\9\1\7 ]] 2023-10-14T00:29:18.735 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:29:18.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:18.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:18.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:18.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:18.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:18.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:18.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:18.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:18.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:18.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:18.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:18.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:18.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:18.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:19.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:19.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:19.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:19.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:19.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:19.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:19.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:19.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:19.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:19.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:21.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:21.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:21.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:21.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:21.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:21.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:21.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:21.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:21.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:21.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:23.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:23.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:23.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:23.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:23.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:23.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:23.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:23.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:23.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:23.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:24.358 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:24.349+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:24.366 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:24.361+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:25.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:25.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:25.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:25.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:25.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:25.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:26.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:26.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:26.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:26.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:26.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:26.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:26.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:26.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:26.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:26.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:27.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:27.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:27.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:27.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:27.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:27.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:27.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:27.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:27.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:27.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:27.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:27.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:27.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:27.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:28.736 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:29:28.736 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:28.737 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:28.737 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:29.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:29.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:29.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:29.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:29.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:29.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:29.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:29.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:30.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3862910917 2023-10-14T00:29:30.260 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:29:30.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243370 2023-10-14T00:29:30.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:30.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3862910917 ]] 2023-10-14T00:29:30.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3862910917 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\2\9\1\0\9\1\7 ]] 2023-10-14T00:29:30.264 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:29:30.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:30.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:30.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:30.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:30.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:30.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:30.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:30.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:30.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:30.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:30.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:30.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:31.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:31.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:31.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:31.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:31.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:31.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:31.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:31.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:31.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:31.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:31.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:31.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:31.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:31.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:31.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:31.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:29:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:29:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:31.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:31.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:32.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:32.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:32.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:32.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:32.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:32.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:32.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:32.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:34.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:34.017+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:34.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:29:34.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:34.041+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:34.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:34.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:34.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:34.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:34.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:34.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:34.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:34.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:34.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:34.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:34.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:34.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:34.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:34.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:34.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:35.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:35.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:35.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:35.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:35.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:35.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:37.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:37.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:37.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:37.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:37.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:37.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:37.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:37.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:37.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:37.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:39.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:39.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:39.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:39.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:39.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:39.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:39.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:39.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:39.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:39.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:40.272 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:40.276 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:40.276 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:29:40.302 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:40.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:29:40.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:40.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:40.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:40.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:40.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:40.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:40.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:40.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:41.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:41.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:41.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:41.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:41.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:41.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:41.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:41.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:41.705 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3862910917 2023-10-14T00:29:41.708 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:29:41.713 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243381 2023-10-14T00:29:41.713 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:41.714 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3862910917 ]] 2023-10-14T00:29:41.714 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3862910917 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\2\9\1\0\9\1\7 ]] 2023-10-14T00:29:41.714 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:29:45.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:45.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:45.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:45.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:45.580 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:45.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:45.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:45.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:45.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:45.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:45.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:45.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:45.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:45.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:45.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:45.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:45.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:45.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:45.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:45.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:45.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:45.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:47.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:47.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:47.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:47.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:47.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:47.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:47.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:47.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:47.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:48.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:48.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:48.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:48.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:48.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:48.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:48.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:48.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:48.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:48.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:48.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:48.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:48.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:48.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:48.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:48.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:48.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:48.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:48.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:48.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:48.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:49.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:29:49.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:49.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:49.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:49.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:49.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:49.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:49.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:49.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:49.741 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.741+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:49.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:49.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:49.800 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.801+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:49.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.801+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:49.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:49.801+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:49.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:49.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:49.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:49.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:49.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:49.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:49.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:49.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:50.033 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:50.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:29:50.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:50.041+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:50.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:50.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:50.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:50.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:50.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:50.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:50.060 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:50.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:50.060 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:50.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:50.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:50.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:50.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:50.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:50.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:50.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:51.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:51.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:51.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:51.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:51.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:51.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:51.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:51.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:51.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:29:51.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:29:51.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:51.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:51.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:51.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:51.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:51.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:51.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:51.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:51.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:51.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:51.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:51.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:51.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:51.716 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:51.716 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:51.719 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:29:51.722 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:51.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:51.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:51.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:51.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:52.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.088+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:29:52.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.088+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:52.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:52.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:29:52.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:52.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:52.107 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:52.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:29:52.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:29:52.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:29:52.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:29:52.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:29:52.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:29:52.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:52.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:52.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:52.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:52.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:52.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:52.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:52.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:52.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:52.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:52.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:52.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:52.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:52.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:53.117 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3862910917 2023-10-14T00:29:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:29:53.127 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243393 2023-10-14T00:29:53.127 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:53.127 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3862910917 ]] 2023-10-14T00:29:53.127 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3862910917 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\2\9\1\0\9\1\7 ]] 2023-10-14T00:29:53.128 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:29:53.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:53.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:53.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:53.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:53.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:53.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:53.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:53.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:53.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:53.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:54.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:54.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:54.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:54.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:54.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:54.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:54.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:54.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:54.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:54.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:55.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:55.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:55.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:55.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:55.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:55.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:55.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:29:55.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:29:55.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:55.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:55.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:55.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:55.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:55.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:55.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:56.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:56.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:56.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:56.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:56.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:56.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:56.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:56.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:56.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:56.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:56.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:56.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:56.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:56.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:56.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:57.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:57.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:57.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:57.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:57.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:57.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:57.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:57.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:57.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:57.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:58.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:58.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:58.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:58.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:58.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:29:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:58.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:29:58.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:29:58.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:29:58.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:58.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:29:59.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:59.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:29:59.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:59.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:29:59.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:59.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:29:59.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:29:59.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:01.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:30:01.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:01.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:01.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:01.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:01.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:01.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:01.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:01.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:01.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:01.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:01.224+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:01.255 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:01.252+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:02.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:30:02.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:02.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:02.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:02.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:02.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:02.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:02.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:02.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:30:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:03.131 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:04.388 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:30:04.389 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:30:04.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243404 2023-10-14T00:30:04.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:04.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2938574976 ]] 2023-10-14T00:30:04.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2938574976 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\8\6\2\9\1\0\9\1\7 ]] 2023-10-14T00:30:04.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2938574976 2023-10-14T00:30:04.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:04.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:04.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:04.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:04.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:04.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:04.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:04.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:05.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:05.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:30:05.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:05.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:05.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:05.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:05.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:05.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:05.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:05.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:05.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:05.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:05.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:05.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:05.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:05.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:05.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:05.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:05.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:05.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:05.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:05.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:05.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:05.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:05.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:05.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:05.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:05.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:06.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:06.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:06.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:06.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:06.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:06.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:06.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:06.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:06.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:06.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:06.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:06.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:06.480 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2938574976 until 2023-10-14T01:30:05.832049+0000 (3600 sec) 2023-10-14T00:30:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2938574976 2023-10-14T00:30:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:30:07.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:07.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:07.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:07.941 INFO:tasks.workunit.client.0.smithi138.stderr:listed 32 entries 2023-10-14T00:30:07.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:07.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:07.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:07.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:07.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:07.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:07.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:07.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:08.003 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:30:08.003 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:30:08.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:08.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:08.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:08.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:08.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:08.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:08.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:08.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:08.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:08.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:08.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:09.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:09.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:09.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:09.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:09.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:09.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:09.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:09.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:12.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:12.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:12.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:12.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:12.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:12.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:12.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:12.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:12.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:12.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:12.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:12.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:12.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:12.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:13.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:13.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:13.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:13.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:13.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:13.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:13.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:13.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:13.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:13.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:13.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:13.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:14.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:14.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:14.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:14.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:14.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:14.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:14.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:14.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:14.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:14.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:14.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:14.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:14.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:14.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:15.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:15.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:15.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:15.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:15.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:15.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:15.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:15.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:15.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:15.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:15.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:15.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:15.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:15.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:30:15.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:15.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:15.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:15.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:15.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:15.644+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:15.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:15.652+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:15.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:15.652+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:15.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:15.652+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:16.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:16.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:16.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:16.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:16.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:16.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:16.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:16.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:16.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:16.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:16.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:16.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:16.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:16.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:16.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:16.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:16.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:17.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:17.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:17.427 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:17.428+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:17.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:17.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:17.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:17.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:17.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:17.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:17.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:30:17.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:17.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:17.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:17.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:17.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:17.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:17.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:17.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:17.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:18.029 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:30:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:18.042 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:18.051 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:18.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:18.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:18.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:18.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:18.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:18.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:18.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:18.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:18.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:19.389 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:30:19.396 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:30:19.398 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243419 2023-10-14T00:30:19.398 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:19.398 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2938574976 ]] 2023-10-14T00:30:19.398 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2938574976 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\3\8\5\7\4\9\7\6 ]] 2023-10-14T00:30:19.399 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:30:19.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:19.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:19.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:19.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:19.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:19.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:19.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:19.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:19.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:19.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:19.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:19.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:22.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:22.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:22.600 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:22.600+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:22.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:22.600+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:22.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:22.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:22.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:22.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:22.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:22.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:23.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:23.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:23.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:23.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:23.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:23.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:23.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:23.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:23.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:23.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:23.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:23.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:23.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:23.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:23.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:23.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:23.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:23.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:24.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:24.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:24.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:24.048+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:24.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:24.048+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:24.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:24.048+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:25.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:25.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:25.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:25.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:25.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:25.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:25.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:25.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:25.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:25.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:25.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:25.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:26.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:26.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:26.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:26.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:26.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:26.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:26.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:26.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:26.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:26.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:26.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:26.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:27.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:27.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:27.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:27.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:27.509 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:27.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:27.509 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:27.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:27.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:27.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:27.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:27.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:27.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:27.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:27.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:27.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:27.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:27.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:29.405 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:29.406 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:29.411 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:29.421 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:30:29.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:29.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:30:29.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:29.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:29.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:29.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:29.975 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:29.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:30.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:30.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:30.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:30.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:30.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:30.886 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:30:30.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:30.887 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:30:30.888 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243430 2023-10-14T00:30:30.888 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:30.888 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2938574976 ]] 2023-10-14T00:30:30.888 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2938574976 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\3\8\5\7\4\9\7\6 ]] 2023-10-14T00:30:30.889 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:30:30.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:30.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:30.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:31.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.300+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:30:31.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.300+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:31.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:31.314 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.312+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:31.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:31.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:31.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:31.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:31.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:31.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:31.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:31.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:31.685 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:31.698 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.696+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:31.699 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.696+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:31.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:31.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:31.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:31.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:30:31.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:31.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:31.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:31.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:31.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:31.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:31.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:31.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.824+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:31.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.824+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:31.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:31.824+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:33.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:33.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:33.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:33.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:33.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:33.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:33.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:33.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:33.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:33.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:33.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:33.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:33.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:33.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:33.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:33.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:33.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:33.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:33.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:33.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:33.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:33.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:33.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:34.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:34.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:30:34.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:34.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:34.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:34.135+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:34.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:34.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:34.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:34.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:34.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:34.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:34.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:34.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:37.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:37.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:37.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:37.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:37.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:37.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:37.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:37.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:37.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:37.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:37.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:37.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:37.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:37.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:38.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:38.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:38.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:38.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:38.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:38.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:38.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:38.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:38.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:38.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:40.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:40.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:40.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:40.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:40.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:40.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:40.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:40.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:40.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:40.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:40.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:40.267+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:40.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:40.279+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:40.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:40.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:40.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:40.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:40.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:40.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:40.903 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:30:40.904 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:40.905 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:40.908 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:42.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:42.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:42.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:42.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:42.360 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:30:42.360 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:30:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243442 2023-10-14T00:30:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2938574976 ]] 2023-10-14T00:30:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2938574976 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\3\8\5\7\4\9\7\6 ]] 2023-10-14T00:30:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:30:42.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:42.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:42.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:42.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:42.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:42.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:45.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:45.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:45.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:45.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:45.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:45.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:45.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:45.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:45.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:45.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:45.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:45.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:46.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:46.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:46.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:46.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:46.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:46.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:46.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:46.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:47.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:47.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:47.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:47.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:47.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:47.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:47.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:47.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:47.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:47.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:47.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:47.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:48.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:48.199+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:48.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:48.199+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:48.208 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:48.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:48.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:48.211+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:48.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:48.211+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:48.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:48.211+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:49.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:49.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:49.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:49.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:49.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:49.191+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:49.196 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:49.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:49.197 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:49.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:49.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:49.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:49.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:49.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:49.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:49.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:49.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:49.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:49.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:49.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:49.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:49.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:49.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:49.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:51.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:51.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:51.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:51.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:51.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:51.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:51.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:51.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:51.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:51.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:51.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:51.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:51.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:51.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:51.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:52.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:52.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:52.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:52.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:52.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:52.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:52.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:52.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:52.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:52.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:52.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:52.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:52.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:52.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:52.362 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:52.363 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:30:52.369 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:52.370 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:53.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:53.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:53.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:53.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:53.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:53.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:53.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:53.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:53.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:53.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:53.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:53.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:53.831 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:30:53.831 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:30:53.833 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243453 2023-10-14T00:30:53.833 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:53.833 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2938574976 ]] 2023-10-14T00:30:53.833 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2938574976 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\3\8\5\7\4\9\7\6 ]] 2023-10-14T00:30:53.833 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:30:54.504 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:54.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:54.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.515+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:54.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.515+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:54.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.515+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:54.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:30:54.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:54.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:54.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:54.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:54.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:54.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:54.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:56.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:56.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:56.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:56.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:56.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:56.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:56.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:56.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:57.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:57.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:57.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:57.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:57.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:57.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:57.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:57.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:57.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:57.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:57.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:58.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:58.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:30:58.836 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:58.835+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:30:58.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:58.843+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:30:58.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:58.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:58.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:58.851+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:30:58.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:58.851+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:30:58.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:30:58.851+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:30:58.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:58.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:58.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:58.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:58.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:58.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:58.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:58.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:58.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:58.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:58.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:58.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:30:59.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:59.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:59.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:59.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:30:59.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:30:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:30:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:30:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:30:59.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:30:59.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:30:59.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:30:59.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:00.005 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:30:59.999+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.006 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:30:59.999+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8e17d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.007+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.007+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f85d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.027+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.027+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f85c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.031+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.031+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f84980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.039+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.039+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928c600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.043 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.043+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.043+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9358080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.043+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.045 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.043+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9359080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.047 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.047+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.047 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.047+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9358400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.049 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.047+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.050 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.047+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9358600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.051+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.051+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9358780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.055+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.055+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9358980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.055+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.058 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.055+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.079 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.079+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.079+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9359c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.095 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.095+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.095 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.095+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9359e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.107+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.107+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.107+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.108 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.107+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.132 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.133 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:31:00.131+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:31:00.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:00.135+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:00.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:00.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:00.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:00.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:00.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:00.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:00.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:00.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:00.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:00.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:00.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:00.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:00.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:00.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:00.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:00.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:00.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:00.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:01.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:01.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:01.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:01.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:01.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:01.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:01.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:01.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:02.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:02.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:02.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:02.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:02.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:02.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:02.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:02.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:02.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:03.835 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:31:03.835 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:03.835 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:03.835 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:05.236 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:31:05.237 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:31:05.238 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243465 2023-10-14T00:31:05.238 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:05.238 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2938574976 ]] 2023-10-14T00:31:05.238 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2938574976 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\3\8\5\7\4\9\7\6 ]] 2023-10-14T00:31:05.238 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:31:06.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:06.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:06.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:06.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:06.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:06.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:06.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:06.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:06.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:06.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:07.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:07.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:07.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:07.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:07.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:07.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:07.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:07.345 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:07.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:07.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:31:07.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:07.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:07.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:07.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:07.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:07.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:07.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:07.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:07.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:07.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:07.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:07.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:07.399+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:07.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:07.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:07.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:07.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:07.406 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:07.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:09.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:09.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:09.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:09.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:09.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:09.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:09.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:09.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:09.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:09.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:09.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:09.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:09.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:09.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:09.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:09.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:10.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:10.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:10.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:10.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:11.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:11.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:11.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:11.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:11.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:11.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:11.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:11.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:11.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:11.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:11.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:11.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:11.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:11.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:11.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:11.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:11.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:11.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:11.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:11.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:11.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:11.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:11.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:11.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:11.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:11.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:12.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:12.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:12.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:12.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:12.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:12.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:12.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:12.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:12.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:12.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:12.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:12.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:12.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:12.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:13.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:13.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:13.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:13.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:13.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:13.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:14.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:14.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:14.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:14.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:14.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:14.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:14.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:14.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:15.240 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:15.245 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:15.247 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:15.247 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:31:16.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:16.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:16.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:16.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:16.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:16.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:16.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:16.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:16.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:16.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:16.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:16.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:16.916 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2938574976 2023-10-14T00:31:16.919 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:31:16.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243476 2023-10-14T00:31:16.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:16.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2938574976 ]] 2023-10-14T00:31:16.921 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2938574976 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\3\8\5\7\4\9\7\6 ]] 2023-10-14T00:31:16.921 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:31:18.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:18.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:18.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:18.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:18.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:18.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:18.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:18.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:18.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:18.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:18.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:18.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:18.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:18.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:18.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:18.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:18.785 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:18.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:18.785 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:18.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:18.785 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:18.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:19.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:19.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:19.124 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:19.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:19.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:19.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:19.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:19.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:19.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:19.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:19.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:19.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:19.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:19.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:19.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:19.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:19.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:19.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:19.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:19.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:19.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:19.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:19.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:21.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:21.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:21.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:21.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:21.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:21.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:21.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:21.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:21.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:21.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:21.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:21.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:22.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:22.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:22.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:22.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:22.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:22.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:22.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:22.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:22.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:22.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:22.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:22.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:22.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:22.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:24.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:24.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:24.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:24.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:24.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:24.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:24.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:24.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:24.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:24.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:24.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:24.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:26.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:26.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:26.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:26.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:26.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:26.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:26.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:26.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:26.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:26.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:26.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:26.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:26.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:26.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:26.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:26.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:26.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:26.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:26.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:26.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:26.937 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:26.948 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:26.958 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:31:26.961 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:28.388 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/112034370 2023-10-14T00:31:28.389 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:31:28.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243488 2023-10-14T00:31:28.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:28.390 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/112034370 ]] 2023-10-14T00:31:28.391 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/112034370 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\3\8\5\7\4\9\7\6 ]] 2023-10-14T00:31:28.391 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/112034370 2023-10-14T00:31:29.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:29.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:29.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:29.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:29.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:29.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:29.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:29.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:29.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:29.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:29.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:29.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:29.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:29.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:29.630 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:29.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:29.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:29.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:29.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:29.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:29.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:29.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:29.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:29.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:29.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:29.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:30.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:30.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:30.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:30.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:30.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:30.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:30.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:30.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:30.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:30.997 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/112034370 until 2023-10-14T01:31:30.385396+0000 (3600 sec) 2023-10-14T00:31:31.081 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:31:31.082 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/112034370 2023-10-14T00:31:31.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:31.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:31.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:31.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:31.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:31.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:31.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:31.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:32.208 INFO:tasks.workunit.client.0.smithi138.stderr:listed 33 entries 2023-10-14T00:31:32.258 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/112034370 2023-10-14T00:31:32.258 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:31:32.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:32.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:32.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:32.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:32.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:32.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:32.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:32.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:34.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:34.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:34.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:34.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:34.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:34.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:34.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:31:34.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:31:34.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:34.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:34.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:34.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:34.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:34.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:34.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:34.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:34.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:34.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:34.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:34.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:35.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:34.994+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:35.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:35.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:35.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.414+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:31:35.419 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.418+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:35.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:35.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:35.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:35.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:35.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:35.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:35.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:35.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:35.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:35.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:35.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:35.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:35.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.442+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:35.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:35.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:35.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:35.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:35.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:36.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:36.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:36.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:36.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:36.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:36.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:36.260 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:31:36.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:36.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:36.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:36.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:36.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:36.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:37.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:31:37.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:31:37.871 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.870+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:37.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:37.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:37.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:37.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:37.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:37.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:38.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:38.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:39.709 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:39.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:31:39.709 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:39.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:39.725 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:39.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:42.271 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:31:42.272 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:42.277 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:42.289 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:42.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:42.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:42.671 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:42.670+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:42.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:42.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:42.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:42.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:42.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:42.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:42.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:42.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:43.706 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/112034370 2023-10-14T00:31:43.706 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:31:43.707 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243503 2023-10-14T00:31:43.707 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:43.707 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/112034370 ]] 2023-10-14T00:31:43.707 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/112034370 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\0\3\4\3\7\0 ]] 2023-10-14T00:31:43.707 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:31:43.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:43.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:43.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:43.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:43.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:43.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:43.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:43.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:43.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:44.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:44.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:31:44.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:44.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:44.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:44.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:44.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:44.910+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:44.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:44.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:44.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:44.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:44.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:44.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:45.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:45.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:45.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:45.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:46.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:46.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:46.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:46.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:46.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:46.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:48.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:48.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:48.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:48.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:48.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:48.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:48.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:48.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:48.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:48.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:48.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:48.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:49.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:49.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:49.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:49.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:49.260 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:49.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:49.279 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:49.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:49.279 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:49.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:49.279 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:49.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:50.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:50.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:50.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:50.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:53.724 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:31:53.727 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:53.741 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:53.741 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:54.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:54.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:54.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:54.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:54.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:54.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:54.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:54.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:54.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:31:54.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:54.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:54.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:54.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:54.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:54.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:54.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:54.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:55.400 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/112034370 2023-10-14T00:31:55.400 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:31:55.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243515 2023-10-14T00:31:55.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:55.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/112034370 ]] 2023-10-14T00:31:55.402 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/112034370 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\0\3\4\3\7\0 ]] 2023-10-14T00:31:55.402 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:31:56.363 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:56.358+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:56.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:56.366+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:56.378 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:56.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:56.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:56.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:56.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:56.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:56.401 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:56.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:57.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:57.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:57.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:57.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:57.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:57.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:57.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:57.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:57.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:57.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:57.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:58.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:58.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:58.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:58.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:58.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:58.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:58.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:58.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:58.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:58.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:58.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:58.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:59.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:59.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:31:59.158 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:59.154+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:31:59.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:59.154+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:31:59.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:59.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:31:59.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:59.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:31:59.202 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:59.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:31:59.202 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:59.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:31:59.202 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:31:59.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:31:59.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:31:59.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:31:59.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:31:59.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:31:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:31:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:31:59.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:31:59.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.018+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.018+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8e17480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.022+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.022+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.026+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.030+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8deaa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.030+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.030+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.046+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.046+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8ae3880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.050+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.050+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9382b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.082 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.082 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.078+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.113 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.113 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.110+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9383a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.115 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:32:00.110+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:32:00.140 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:00.138+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:00.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:00.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:00.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:00.170+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:01.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:01.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:01.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:01.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:01.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:01.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:01.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:01.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:01.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:01.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:02.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:32:02.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:02.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:02.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:02.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:02.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:02.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:02.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:02.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:02.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:04.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:32:04.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:04.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:04.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:04.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:04.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:04.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:04.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:04.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:05.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:05.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:05.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:05.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:05.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:05.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:05.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:05.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:05.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:05.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:05.408 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:05.408 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:32:05.409 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:05.421 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/112034370 2023-10-14T00:32:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:32:06.784 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243526 2023-10-14T00:32:06.785 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:06.785 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/112034370 ]] 2023-10-14T00:32:06.785 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/112034370 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\0\3\4\3\7\0 ]] 2023-10-14T00:32:06.785 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:32:08.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:08.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:08.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:08.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:08.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:08.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:08.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:08.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:08.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:08.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:08.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:08.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:08.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:10.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:10.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:32:10.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:10.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:10.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:10.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:10.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:10.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:10.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:10.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:12.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:12.725+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:12.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:12.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:12.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:12.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:12.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:12.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:12.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:12.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:12.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:12.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:13.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:13.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:13.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:13.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:13.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:13.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:13.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:13.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:13.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:14.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:14.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:14.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:14.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:14.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:14.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:15.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:15.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:15.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:15.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:15.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:15.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:15.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:15.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:15.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:15.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:15.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:15.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:15.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:15.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:15.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:32:15.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:15.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:15.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:15.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:15.559 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:15.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:15.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:15.597+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:15.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:15.597+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:15.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:15.597+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:15.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:15.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:15.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:15.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:15.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:16.792 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:16.792 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:32:16.795 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:16.801 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:17.449 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:32:17.445+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:32:17.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:17.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:17.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:17.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:17.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:17.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:17.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:17.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:17.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:17.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:17.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:17.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:17.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:17.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:18.261 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/112034370 2023-10-14T00:32:18.261 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:32:18.262 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243538 2023-10-14T00:32:18.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:18.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/112034370 ]] 2023-10-14T00:32:18.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/112034370 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\0\3\4\3\7\0 ]] 2023-10-14T00:32:18.263 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:32:19.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:19.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:19.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:19.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:19.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:19.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:19.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:19.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:19.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:19.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:19.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:20.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:20.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:20.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:20.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:20.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:20.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:20.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:20.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:20.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:20.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:20.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:20.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:20.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:20.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:22.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:22.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:22.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:22.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:22.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:22.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:22.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.897+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:22.931 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.929+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:22.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.929+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:22.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:22.929+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:23.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:23.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:26.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:26.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:26.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:26.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:26.837 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:26.833+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:26.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:26.897+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:26.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:26.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:26.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:26.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:26.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:26.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:26.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:26.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:26.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:26.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:26.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:26.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:26.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:26.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:26.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:26.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:26.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:26.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:27.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:27.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:32:27.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:27.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:27.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:27.737+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:27.755 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:27.753+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:27.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:27.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:27.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:27.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:28.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:28.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:28.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:28.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:28.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:28.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:28.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:28.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:28.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:28.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:28.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:28.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:28.266 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:28.269 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:28.270 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:28.274 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:32:29.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:29.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:29.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:29.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:29.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:29.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:29.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:32:29.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:32:29.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:29.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:29.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:29.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:29.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:29.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:29.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:29.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:29.644 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/112034370 2023-10-14T00:32:29.644 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:32:29.645 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243549 2023-10-14T00:32:29.645 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:29.645 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/112034370 ]] 2023-10-14T00:32:29.645 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/112034370 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\0\3\4\3\7\0 ]] 2023-10-14T00:32:29.646 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:32:29.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:29.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:29.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:29.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:29.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:29.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:29.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:32:29.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:29.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:29.871 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:29.869+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:29.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:29.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:30.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:30.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:31.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:31.041+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:31.045 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:31.045+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:31.055 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:31.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:31.055 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:31.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:31.055 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:31.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:31.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:31.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:31.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:31.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:31.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:31.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:31.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:31.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:31.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:31.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:31.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:31.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:31.423 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:32:31.421+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:32:33.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:33.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:33.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:33.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:33.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:33.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:33.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:33.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:33.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:33.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:33.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:33.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:34.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:34.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:34.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:34.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:34.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:34.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:34.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:34.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:34.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:34.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:34.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:34.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:34.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:34.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:34.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:34.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:37.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:37.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:37.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:37.945+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:37.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:37.945+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:37.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:37.945+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:38.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:38.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:38.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:38.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:38.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:38.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:38.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:38.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:38.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:38.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:38.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:38.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:38.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:38.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:38.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:39.043 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.041+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:32:39.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:39.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:39.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.085+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:39.089 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:39.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:39.322 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.321+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:32:39.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.353+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:39.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:39.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:39.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:39.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:39.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:39.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:39.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:39.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:39.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:39.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:39.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:39.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:39.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:39.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:39.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:39.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:39.647 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:32:39.648 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:39.648 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:39.657 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:40.934 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1128175262 2023-10-14T00:32:40.934 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:32:40.935 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243560 2023-10-14T00:32:40.935 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:40.935 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1128175262 ]] 2023-10-14T00:32:40.936 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1128175262 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\0\3\4\3\7\0 ]] 2023-10-14T00:32:40.936 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/1128175262 2023-10-14T00:32:42.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:42.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:42.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:42.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:42.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:42.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:42.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:42.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:42.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:42.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:42.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:42.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:42.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:42.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:42.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:42.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:43.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:43.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:43.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:43.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:43.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:43.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:43.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:43.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:43.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:43.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:43.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:43.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:43.573 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/1128175262 until 2023-10-14T01:32:42.931657+0000 (3600 sec) 2023-10-14T00:32:43.654 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/1128175262 2023-10-14T00:32:43.659 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:32:43.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:43.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:44.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:44.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:44.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:44.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:44.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:44.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:44.755 INFO:tasks.workunit.client.0.smithi138.stderr:listed 34 entries 2023-10-14T00:32:44.795 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/1128175262 2023-10-14T00:32:44.796 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:32:45.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:45.037+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:32:45.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:45.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:32:45.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:45.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:32:45.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:32:45.057+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:32:45.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:45.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:45.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:45.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:45.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:45.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:45.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:45.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:45.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:46.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:46.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:46.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:46.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:46.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:46.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:46.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:46.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:46.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:46.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:46.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:46.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:46.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:46.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:49.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:49.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:49.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:49.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:49.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:49.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:49.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:49.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:49.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:49.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:51.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:51.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:51.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:51.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:51.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:51.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:51.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:51.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:51.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:51.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:51.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:51.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:51.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:51.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:53.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:53.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:53.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:53.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:53.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:53.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:53.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:53.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:53.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:53.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:53.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:53.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:54.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:54.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:54.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:54.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:54.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:54.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:54.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:54.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:54.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:54.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:32:54.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:54.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:32:54.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:54.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:32:54.806 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:54.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:54.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:32:54.830 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:56.187 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1128175262 2023-10-14T00:32:56.188 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:32:56.189 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243576 2023-10-14T00:32:56.189 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:56.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1128175262 ]] 2023-10-14T00:32:56.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1128175262 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\8\1\7\5\2\6\2 ]] 2023-10-14T00:32:56.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:32:59.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:59.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:32:59.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:59.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:32:59.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:59.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:32:59.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:59.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:32:59.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:59.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:32:59.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:32:59.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:00.002 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.000+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.002 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.000+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9109800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.024+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.023 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.024+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9358680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8e17a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9359b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d89c0c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f85280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd8f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.028+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f84e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.032+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.032+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.040+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.040+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.044+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.044+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8deb480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.048+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.048+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8deb200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.056+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.056+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9382b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.056+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.056+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9383980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9383780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.086 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d945e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.087 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9383600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.087 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9383c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.087 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.084+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d945ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.092+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.095 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:33:00.092+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d945e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:33:00.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:00.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:00.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:00.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:00.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:00.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:00.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:00.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:00.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:00.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:00.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:00.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:01.026 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:01.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:01.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:01.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:01.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:01.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:01.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:01.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:01.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:01.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:01.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:01.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:01.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:01.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:01.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:03.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:03.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:03.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:03.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:03.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:03.356+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:03.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:03.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:03.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:03.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:03.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:03.412+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:03.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:03.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:03.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:03.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:03.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:03.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:03.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:03.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:03.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:04.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:04.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:04.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:04.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:04.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:04.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:04.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:04.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:04.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:04.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:04.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:04.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:05.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:05.112+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:05.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:05.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:05.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:05.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:05.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:05.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:05.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:05.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:05.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:05.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:05.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:05.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:06.191 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:06.192 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:06.202 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:06.206 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:33:06.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:06.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:06.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:06.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:06.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:06.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:06.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:06.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:07.623 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1128175262 2023-10-14T00:33:07.624 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:33:07.624 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243587 2023-10-14T00:33:07.625 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:07.625 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1128175262 ]] 2023-10-14T00:33:07.625 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1128175262 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\8\1\7\5\2\6\2 ]] 2023-10-14T00:33:07.625 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:33:08.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:08.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:08.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:08.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:08.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:08.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:08.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:08.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:08.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:08.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:08.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:08.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:10.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:10.360+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:10.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:10.360+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:10.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:10.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:10.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:10.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:10.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:10.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:11.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:11.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:11.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:11.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:11.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:11.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:11.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:11.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:11.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:12.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:12.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:12.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:12.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:12.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:12.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:12.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:12.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:12.111 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.112+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:12.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.112+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:12.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.164+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:12.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:12.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:12.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:12.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:12.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:12.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:12.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.652+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:12.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.716+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:12.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:12.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:12.730 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:12.781 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:12.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:12.815 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:12.815 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:12.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:12.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:14.766 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:14.764+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:14.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:14.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:14.795 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:14.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:14.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:14.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:14.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:14.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:14.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:14.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:14.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:14.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:14.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:14.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:14.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:14.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:15.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:15.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:15.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:15.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:15.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:15.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:15.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:15.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:15.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:15.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:15.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:15.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:16.397 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:16.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:16.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:16.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:16.419 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:16.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:16.437 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:16.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:16.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:16.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:16.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:16.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:16.446 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:16.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:16.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:16.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:16.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:16.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:16.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:16.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:16.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:16.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:16.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:16.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:16.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:16.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:17.632 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:17.635 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:33:17.635 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:17.654 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:18.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:18.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:18.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:18.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:18.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:18.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:19.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1128175262 2023-10-14T00:33:19.008 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:33:19.009 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243599 2023-10-14T00:33:19.009 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:19.009 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1128175262 ]] 2023-10-14T00:33:19.010 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1128175262 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\8\1\7\5\2\6\2 ]] 2023-10-14T00:33:19.010 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:33:19.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:19.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:19.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:19.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:19.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:19.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:19.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:19.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:19.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:19.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:19.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:19.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:20.976 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:20.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:20.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:20.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:20.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:20.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:22.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:22.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:22.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:22.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:22.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:22.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:22.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:22.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:22.388 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:22.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:22.418 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:22.416+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:22.418 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:22.416+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:22.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:22.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:22.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:22.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:24.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:24.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:24.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:24.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:24.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:24.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:24.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:24.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:24.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:24.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:33:24.617 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:24.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:24.624 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:24.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:24.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:24.640+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:24.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:24.644+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:24.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:24.644+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:24.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:24.644+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:25.066 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:25.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:25.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:25.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:25.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:25.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:25.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:25.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:25.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:25.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:26.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:26.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:26.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:26.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:26.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:26.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:26.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:26.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:26.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:26.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:26.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:26.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:26.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:26.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:26.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:26.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:27.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:27.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:27.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:27.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:27.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:27.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:27.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:27.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:27.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:28.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:28.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:28.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:28.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:28.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:28.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:28.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:28.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:28.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:28.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:28.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:28.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:28.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:28.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:29.011 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:29.012 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:29.012 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:29.024 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:33:29.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:29.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:29.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:29.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:29.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:29.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:29.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:29.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:29.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:30.252 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1128175262 2023-10-14T00:33:30.252 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:33:30.253 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243610 2023-10-14T00:33:30.253 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:30.253 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1128175262 ]] 2023-10-14T00:33:30.254 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1128175262 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\8\1\7\5\2\6\2 ]] 2023-10-14T00:33:30.254 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:33:31.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:31.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:31.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:31.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:31.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:31.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:31.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:31.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:31.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:31.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:31.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:31.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:31.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:31.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:32.609 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:32.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:32.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:32.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:32.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:32.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:32.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:32.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:32.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:32.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:32.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:32.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:32.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:32.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:33.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:33.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:34.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:34.044+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:34.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:34.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:34.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:34.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:34.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:34.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:34.060+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:34.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:34.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:34.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:34.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:34.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:34.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:34.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:34.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:34.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:34.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:34.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:34.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:34.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:34.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:34.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:35.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:35.428+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:35.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:35.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:35.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:35.480+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:35.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:35.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:35.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:35.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:35.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:35.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:36.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:36.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:36.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:36.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:36.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:36.192+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:36.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:36.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:36.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:36.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:36.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:33:36.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:36.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:36.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:36.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:36.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:36.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:36.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:36.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:36.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:37.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:37.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:37.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:37.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:37.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:37.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:37.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:37.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:37.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:37.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:38.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:38.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:38.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:38.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:38.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:38.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:38.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:38.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:38.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:38.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:39.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:39.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:39.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:39.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:39.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:39.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:39.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:39.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:39.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:39.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:40.257 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:33:40.257 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:40.258 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:40.265 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:40.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:40.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:40.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:40.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:41.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:41.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:41.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:41.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:41.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:41.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:41.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:41.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:41.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:41.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:41.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:41.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:41.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:41.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:41.827 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/1128175262 2023-10-14T00:33:41.834 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:33:41.843 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243621 2023-10-14T00:33:41.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:41.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/1128175262 ]] 2023-10-14T00:33:41.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/1128175262 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\8\1\7\5\2\6\2 ]] 2023-10-14T00:33:41.844 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:33:42.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:42.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:42.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:42.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:42.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:42.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:42.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:42.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:42.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:42.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:42.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:42.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:43.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:43.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:43.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:43.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:43.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:43.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:43.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:43.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:43.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:43.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:43.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:43.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:45.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:33:45.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:45.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:45.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:45.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:45.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:45.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:45.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:45.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:46.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:46.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:46.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:46.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:46.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:46.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:46.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:46.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:46.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:46.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:46.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:46.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:46.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:46.872+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:46.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:46.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:46.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:46.952+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:46.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:46.952+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:47.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:47.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:47.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:47.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:47.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:47.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:47.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:47.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:49.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:49.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:49.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:49.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:49.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:49.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:49.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:50.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:50.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:51.345 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:51.343+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:51.363 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:51.363+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:51.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:51.379+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:33:51.383 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:51.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:51.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:51.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:33:51.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:51.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:33:51.392 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:51.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:33:51.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:51.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:51.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:51.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:51.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:33:51.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:33:51.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:51.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:51.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:51.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:51.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:51.860 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:33:51.860 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:51.867 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:51.870 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:53.181 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2872413020 2023-10-14T00:33:53.186 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:33:53.187 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243633 2023-10-14T00:33:53.188 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:53.188 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2872413020 ]] 2023-10-14T00:33:53.188 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2872413020 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\1\1\2\8\1\7\5\2\6\2 ]] 2023-10-14T00:33:53.188 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2872413020 2023-10-14T00:33:54.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:54.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:54.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:54.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:54.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:54.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:54.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:54.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:54.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:55.582 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2872413020 until 2023-10-14T01:33:54.897155+0000 (3600 sec) 2023-10-14T00:33:55.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:33:55.634 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2872413020 2023-10-14T00:33:56.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:33:56.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:56.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:56.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:56.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:56.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:56.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:56.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:56.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:33:56.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:56.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:56.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:56.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:56.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:56.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:56.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:56.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:56.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:56.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:33:56.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:56.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:33:56.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:33:56.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:33:56.780 INFO:tasks.workunit.client.0.smithi138.stderr:listed 35 entries 2023-10-14T00:33:56.846 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2872413020 2023-10-14T00:33:56.847 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:33:56.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:56.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:56.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:56.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:56.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:56.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:33:57.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:57.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:57.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:57.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:57.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:57.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:57.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:57.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:57.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:57.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:58.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:33:58.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:33:58.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:33:58.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:33:58.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:33:58.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:33:58.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:33:58.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:33:58.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:00.008 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.007+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.008 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.007+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f84100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.015+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.015 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.015+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9359f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.027 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.027+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.027 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.027+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.047 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.047+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.047 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.047+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9383500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.071+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945f000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945ee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945ec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945f280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.075+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9383e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945f480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945fa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.095 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.095 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.091+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.099 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.099+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.099 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.099+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.103+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:00.103 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:34:00.103+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9460080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:34:02.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:02.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:04.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:04.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:34:04.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:04.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:04.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:04.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:34:04.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:04.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:04.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:04.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:04.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:04.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:04.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:04.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:04.097 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:04.095+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:04.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:04.099+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:04.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:04.099+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:04.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:04.099+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:04.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:04.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:04.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:04.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:04.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:04.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:05.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:05.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:05.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:05.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:05.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:05.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:05.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:05.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:05.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:05.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:06.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:06.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:06.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:06.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:06.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:06.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:06.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:06.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:06.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:06.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:34:06.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:06.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:06.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:06.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:06.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:06.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:06.865 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:06.866 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:06.868 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:06.869 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:34:07.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:07.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:07.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:07.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:07.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:07.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:07.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:07.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:07.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:08.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:08.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:08.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:08.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:08.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:08.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:08.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:08.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:08.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:08.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:08.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:08.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:08.175 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2872413020 2023-10-14T00:34:08.175 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:34:08.179 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243648 2023-10-14T00:34:08.179 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:08.179 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2872413020 ]] 2023-10-14T00:34:08.180 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2872413020 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\7\2\4\1\3\0\2\0 ]] 2023-10-14T00:34:08.180 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:34:10.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:10.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:10.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:10.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:10.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:10.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:10.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:10.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:11.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:11.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:11.593 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:11.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:13.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:13.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:13.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:13.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:13.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:13.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:13.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:13.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:13.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:13.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:13.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:34:13.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:34:13.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:13.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:13.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:13.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:13.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:13.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:13.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:13.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:14.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:34:14.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:14.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:14.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:14.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:14.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:14.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:14.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:14.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:15.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:15.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:15.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:15.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:15.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:15.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:15.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:15.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:15.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:15.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:15.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:15.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:15.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:15.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:15.992 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:15.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:16.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:16.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:16.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:16.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:16.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:16.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:16.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:16.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:16.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:16.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:18.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:18.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:18.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:18.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:18.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:34:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:18.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:34:18.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:18.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:18.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:18.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:18.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:18.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:18.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:18.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:18.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:18.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:34:18.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:18.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:18.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:18.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:18.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:18.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:18.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:18.819+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:18.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:18.819+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:18.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:18.819+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:19.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:19.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:19.580 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2872413020 2023-10-14T00:34:19.616 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:34:19.617 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243659 2023-10-14T00:34:19.618 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:19.618 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2872413020 ]] 2023-10-14T00:34:19.618 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2872413020 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\7\2\4\1\3\0\2\0 ]] 2023-10-14T00:34:19.618 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:34:19.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:19.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:19.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:19.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:19.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:19.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:19.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:19.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:19.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:19.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:21.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:21.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:21.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:21.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:21.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:21.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:21.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:21.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:21.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:21.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:21.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:21.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:22.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:22.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:22.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:22.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:22.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:22.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:22.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:22.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:22.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:22.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:22.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:22.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:22.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:22.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:34:22.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:22.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:22.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:22.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:22.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:22.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:22.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:22.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:22.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:22.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:22.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:22.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:26.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:26.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:34:26.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:26.851+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:26.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:26.883+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:26.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:26.895+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:27.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:27.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:34:27.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:27.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:27.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:27.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:27.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:27.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:27.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:27.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:27.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:27.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:27.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:27.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:28.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:28.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:28.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:28.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:28.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:28.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:28.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:28.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:29.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:29.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:29.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:29.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:29.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:29.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:29.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:29.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:29.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:29.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:29.630 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:34:29.630 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:29.639 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:29.640 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:30.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:30.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:34:30.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:30.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:30.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:30.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:30.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:30.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:30.668 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2872413020 2023-10-14T00:34:30.681 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:34:30.683 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243670 2023-10-14T00:34:30.683 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:30.684 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2872413020 ]] 2023-10-14T00:34:30.684 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2872413020 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\7\2\4\1\3\0\2\0 ]] 2023-10-14T00:34:30.684 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:34:30.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:30.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:30.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:30.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:30.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:30.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:31.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:31.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:31.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:31.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:31.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:31.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:31.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:31.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:31.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:31.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:32.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:32.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:32.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:32.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:32.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:32.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:32.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:32.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:34.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:34.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:34.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:34.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:34.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:34.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:34.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:34.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:34.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:34.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:34.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:34.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:34.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:34.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:35.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:35.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:34:35.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:35.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:35.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:35.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:35.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:35.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:35.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:35.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:35.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:35.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:35.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:35.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:36.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:36.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:36.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:36.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:36.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:36.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:36.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:36.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:39.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:39.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:39.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:39.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:39.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:39.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:39.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:39.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:39.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:39.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:39.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:40.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:40.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:40.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:40.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:40.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:40.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:40.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:40.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:40.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:40.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:40.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:40.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:40.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:40.685 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:40.686 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:34:40.695 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:40.697 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:42.119 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2872413020 2023-10-14T00:34:42.131 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:34:42.132 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243682 2023-10-14T00:34:42.133 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:42.133 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2872413020 ]] 2023-10-14T00:34:42.133 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2872413020 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\7\2\4\1\3\0\2\0 ]] 2023-10-14T00:34:42.133 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:34:42.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:42.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:42.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:42.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:42.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:42.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:43.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:43.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:43.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:43.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:43.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:43.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:43.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:43.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:46.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:46.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:46.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:46.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:46.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:46.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:46.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:46.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:46.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:46.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:47.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:47.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:47.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:47.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:47.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:47.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:47.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:47.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:47.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:47.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:47.804 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:47.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:34:47.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:47.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:47.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:47.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:47.820 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:47.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:47.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:47.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:47.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:47.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:47.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:47.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:48.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:48.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:48.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:48.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:48.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:48.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:48.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:48.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:48.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:48.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:48.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:48.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:49.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:49.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:34:49.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:49.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:49.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:49.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:49.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:49.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:49.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:49.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:49.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:49.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:49.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:49.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:50.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:50.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:50.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:50.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:51.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:51.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:52.139 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:34:52.139 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:52.139 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:52.150 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:52.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:52.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:52.314 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:52.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:52.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:52.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:52.357 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:52.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:34:52.357 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:52.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:34:53.673 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2872413020 2023-10-14T00:34:53.674 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:34:53.675 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243693 2023-10-14T00:34:53.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:53.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2872413020 ]] 2023-10-14T00:34:53.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2872413020 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\7\2\4\1\3\0\2\0 ]] 2023-10-14T00:34:53.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:34:53.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:53.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:53.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:53.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:55.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:55.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:55.511 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:55.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:34:55.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:55.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:55.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:55.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:34:55.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:55.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:55.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:55.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:55.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:55.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:55.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:55.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:56.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:56.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:56.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:56.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:56.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:56.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:56.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:56.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:56.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:56.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:56.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:56.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:56.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:56.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:56.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:57.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:34:57.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:57.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:57.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:57.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:57.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:57.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:57.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:57.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:57.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:57.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:34:58.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:58.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:34:58.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:34:58.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:34:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:59.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:34:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:59.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:34:59.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:59.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:34:59.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:59.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:34:59.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:34:59.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:00.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:00.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:00.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:00.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:00.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:00.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:01.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:01.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:01.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:01.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:01.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:01.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:01.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:01.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:02.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:02.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:02.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:02.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:02.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:02.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:02.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:02.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:02.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:02.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:02.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:02.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:02.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:02.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:02.871 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:02.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:02.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:02.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:02.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:02.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:02.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:02.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:02.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:02.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:02.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:02.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:03.686 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:03.694 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:03.703 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:03.715 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:35:04.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:04.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:04.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:04.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:04.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:04.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:04.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:04.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:04.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:04.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:04.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:04.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:04.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:04.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:05.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/801064916 2023-10-14T00:35:05.178 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:35:05.179 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243705 2023-10-14T00:35:05.180 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:05.180 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/801064916 ]] 2023-10-14T00:35:05.180 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/801064916 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\8\7\2\4\1\3\0\2\0 ]] 2023-10-14T00:35:05.180 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/801064916 2023-10-14T00:35:05.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:05.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:06.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:06.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:06.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:06.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:06.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:06.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:06.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:06.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:06.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:06.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:06.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:06.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:07.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:07.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:07.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:07.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:07.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:07.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:07.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:07.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:07.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:07.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:07.605 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/801064916 until 2023-10-14T01:35:06.983197+0000 (3600 sec) 2023-10-14T00:35:07.677 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:35:07.677 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/801064916 2023-10-14T00:35:07.873 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:07.870+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:08.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:08.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:08.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:08.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:08.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:08.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:08.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:08.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:08.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:08.748 INFO:tasks.workunit.client.0.smithi138.stderr:listed 36 entries 2023-10-14T00:35:08.820 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/801064916 2023-10-14T00:35:08.820 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:35:10.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:10.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:10.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:10.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:10.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:10.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:10.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:10.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:10.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:10.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:10.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:10.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:10.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:10.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:11.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:11.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:11.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:11.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:11.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:11.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:11.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:11.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:11.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:11.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:11.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:11.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:13.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:13.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:13.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:13.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:13.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:13.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:15.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:15.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:35:16.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:16.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:16.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:16.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:16.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:16.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:16.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:16.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:16.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:16.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:16.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:16.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:16.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:16.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:16.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:16.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:16.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:16.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:16.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:16.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:16.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:16.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:17.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:17.406+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:17.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:17.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:17.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:17.450+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:17.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:17.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:17.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:17.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:17.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:17.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:17.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:17.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:17.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:17.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:17.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:17.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:17.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:17.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:17.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:17.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:18.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:18.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:18.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:18.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:18.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:18.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:18.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:18.851 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:35:18.852 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:18.852 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:18.866 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:18.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:18.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:19.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:19.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:19.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:19.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:19.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:19.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:19.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:19.345 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:19.342+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:19.345 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:19.342+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:19.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:19.366+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:19.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:19.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:19.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:19.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:19.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:19.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:19.414 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:19.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:19.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:35:19.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:19.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:19.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:19.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:19.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:19.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:19.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:19.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:19.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:20.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:20.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:20.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:20.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:20.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:20.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:35:20.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:20.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:20.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:20.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:20.505 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/801064916 2023-10-14T00:35:20.506 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:35:20.507 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243720 2023-10-14T00:35:20.507 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:20.507 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/801064916 ]] 2023-10-14T00:35:20.507 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/801064916 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\8\0\1\0\6\4\9\1\6 ]] 2023-10-14T00:35:20.507 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:35:20.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:20.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:20.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:20.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:20.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:20.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:20.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:20.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:20.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:20.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:21.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:21.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:21.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:21.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:21.269 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:21.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:21.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:21.318+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:21.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:21.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:21.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:21.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:21.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:21.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:22.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:22.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:22.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:22.654+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:22.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:22.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:22.685 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:22.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:22.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:22.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:22.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:22.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:22.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:22.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:23.154 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:23.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:23.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:23.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:23.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:23.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:23.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:23.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:23.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:23.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:23.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:23.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:24.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:24.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:24.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:24.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:24.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:24.854+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:24.907 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:24.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:24.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:24.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:24.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:24.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:25.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:25.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:25.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:25.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:25.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:25.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:25.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:25.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:25.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:25.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:25.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:25.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:25.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:25.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:27.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:27.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:27.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:27.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:27.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:27.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:27.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:27.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:27.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:27.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:27.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:27.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:28.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:28.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:28.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:28.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:28.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:28.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:28.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:28.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:28.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:28.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:28.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:28.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:28.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:28.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:30.510 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:30.515 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:30.518 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:35:30.535 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:31.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:31.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:31.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:31.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:31.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:31.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:31.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:31.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:31.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:31.274+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:31.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:31.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:31.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:31.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:31.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:31.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/801064916 2023-10-14T00:35:31.920 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:35:31.922 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243731 2023-10-14T00:35:31.922 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:31.922 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/801064916 ]] 2023-10-14T00:35:31.923 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/801064916 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\8\0\1\0\6\4\9\1\6 ]] 2023-10-14T00:35:31.923 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:35:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:33.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:33.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:33.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:33.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:33.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:33.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:33.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:36.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:36.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:36.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:36.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:36.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:36.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:36.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:36.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:36.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:36.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:36.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:36.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:37.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:37.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:37.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:37.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:37.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:37.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:37.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:37.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:37.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:37.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:37.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:37.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:37.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:37.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:39.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:39.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:39.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:39.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:39.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:39.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:39.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:39.494+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:39.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:39.494+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:39.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:39.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:39.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:39.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:39.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:39.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:39.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:40.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:40.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:40.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:40.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:40.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:40.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:40.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:40.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:40.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:40.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:41.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:41.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:41.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:41.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:41.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:41.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:41.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:41.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:41.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:41.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:41.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:41.944 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:41.945 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:41.954 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:35:41.961 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:42.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:42.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:42.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:42.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:42.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:42.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:42.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:42.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:43.289 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/801064916 2023-10-14T00:35:43.289 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:35:43.290 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243743 2023-10-14T00:35:43.290 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:43.290 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/801064916 ]] 2023-10-14T00:35:43.290 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/801064916 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\8\0\1\0\6\4\9\1\6 ]] 2023-10-14T00:35:43.291 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:35:44.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:44.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:44.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:44.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:44.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:44.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:44.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:44.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:44.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:44.603 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.597+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:35:44.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.597+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:44.664 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:44.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.677+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:44.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:44.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:44.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.685+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:35:44.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.713+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:44.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.713+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:44.781 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.777+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:44.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:44.777+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:45.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:45.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:45.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:45.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:45.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:45.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:45.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:45.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:45.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:45.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:45.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:45.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:45.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:45.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:45.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:46.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:46.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:46.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:46.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:46.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:46.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:46.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:46.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:46.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:46.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:47.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:47.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:47.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:47.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:53.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:53.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:53.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:53.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:53.304 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:35:53.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:53.306 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:53.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:53.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:53.351 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:53.367 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:53.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:53.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:53.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:53.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:53.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:53.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:53.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:53.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:54.698 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/801064916 2023-10-14T00:35:54.698 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:35:54.699 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243754 2023-10-14T00:35:54.699 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:54.700 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/801064916 ]] 2023-10-14T00:35:54.700 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/801064916 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\8\0\1\0\6\4\9\1\6 ]] 2023-10-14T00:35:54.700 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:35:54.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:54.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:54.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:54.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:54.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:54.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:54.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:54.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:54.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:54.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:55.527 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:55.525+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:55.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:55.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:55.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:55.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:55.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:55.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:56.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:56.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:56.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:56.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:56.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:56.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:56.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:56.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:56.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:56.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:56.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:56.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:56.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:56.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:57.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:57.213+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:35:57.233 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:57.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:35:57.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:57.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:35:57.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:57.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:35:57.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:35:57.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:35:58.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:35:58.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:35:58.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:35:58.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:58.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:58.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:58.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:35:58.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:58.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:59.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:59.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:59.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:35:59.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:35:59.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:59.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:35:59.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:35:59.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:00.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:35:59.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.009+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.012 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.009+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9359a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.009+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.009+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d945e200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.009+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.009+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9383b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.024 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.021+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.025 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.021+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.037 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.033+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.037 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.033+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9383f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.033+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.033+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9460780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.057+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.057+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9460380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.077+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.080 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.077+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9460500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.090 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.085+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.090 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.085+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9460b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.090 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.085+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.085+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9460a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.120 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.117+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.120 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.117+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9460180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.121 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.117+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.121 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.117+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9382f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.135 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.129+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.135 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.129+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9460e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.135 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.133+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.135 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.133+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.136 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.133+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.136 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.133+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.143 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.141+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.143 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.141+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.157+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.157+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.157+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.157+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.163 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.157+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.164 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.157+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.172 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.169+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:00.172 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:00.169+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:01.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:01.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:01.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:01.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:01.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:01.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:02.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:36:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:02.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:02.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.185+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:36:02.194 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:36:02.194 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:02.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.213+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:02.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:02.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:02.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:02.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:02.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:36:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:02.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:02.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:04.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:04.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:04.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:04.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:04.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:04.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:04.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:04.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:04.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:04.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:04.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:04.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:04.701 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:36:04.701 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:04.724 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:04.728 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:05.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:05.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:05.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:05.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:05.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:05.433+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:05.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:05.433+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:05.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:05.433+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:05.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:05.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:05.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:05.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:05.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:05.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:05.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:05.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:05.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:05.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:05.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:05.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:06.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:06.192 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/801064916 2023-10-14T00:36:06.211 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:36:06.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243766 2023-10-14T00:36:06.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:06.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/801064916 ]] 2023-10-14T00:36:06.212 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/801064916 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\8\0\1\0\6\4\9\1\6 ]] 2023-10-14T00:36:06.213 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:36:06.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:06.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:06.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:06.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:06.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:06.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:06.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:06.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:06.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:06.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:06.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:06.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:08.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:08.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:08.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:08.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:09.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:09.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:09.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:09.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:09.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:09.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:09.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:09.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:09.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:09.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:09.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:09.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:10.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:36:10.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:36:10.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:10.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:10.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:10.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:10.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:10.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:10.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:10.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:11.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:11.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:36:11.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:11.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:11.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:11.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:11.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:11.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:11.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:11.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:11.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:11.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:13.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:13.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:13.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:13.545+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:13.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:13.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:13.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:13.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:13.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:13.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:13.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:13.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:14.363 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:14.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:36:14.363 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:14.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:14.376 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:14.377+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:14.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:14.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:14.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:14.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:14.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:14.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:14.418 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:14.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:15.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:15.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:36:15.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:15.301+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:15.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:15.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:15.396 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:15.397+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:15.401 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:15.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:15.401 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:15.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:16.214 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:16.215 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:16.217 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:36:16.240 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:16.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:36:16.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:16.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:16.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:16.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:16.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:16.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:16.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:16.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:17.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:17.245+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:36:17.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:17.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:17.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:17.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:17.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:17.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:17.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:17.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:17.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:17.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:17.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:17.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:17.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:17.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:17.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:17.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:17.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:17.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:17.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:17.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:17.811 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/296162553 2023-10-14T00:36:17.815 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:36:17.833 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243777 2023-10-14T00:36:17.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:17.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/296162553 ]] 2023-10-14T00:36:17.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/296162553 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\8\0\1\0\6\4\9\1\6 ]] 2023-10-14T00:36:17.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/296162553 2023-10-14T00:36:18.410 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:18.409+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:36:18.410 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:18.409+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:18.410 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:18.409+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:18.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:18.677+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:18.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:18.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:18.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:18.729+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:19.668 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.665+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8211140 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.688 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.689+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8210900 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.715 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.717+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8211380 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.790 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.793+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8a87d40 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.800 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.801+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8463800 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.801 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.801+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d859a840 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.801 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.801+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d859ab40 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.802 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.801+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d7e75980 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.802 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.801+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8a86cc0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.803 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.805+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d859b800 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.803 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.805+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d7e74780 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.804 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.805+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8211d40 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:19.804 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:36:19.805+0000 7f0823307640 -1 librbd::api::Mirror: image_info_list: failed to list mirror image statuses: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:36:20.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:20.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:20.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:20.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:20.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:20.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:20.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:20.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:20.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:20.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:20.646 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/296162553 until 2023-10-14T01:36:20.023332+0000 (3600 sec) 2023-10-14T00:36:20.681 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/296162553 2023-10-14T00:36:20.683 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:36:21.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:21.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:21.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:21.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:21.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:21.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:21.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:21.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:21.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:21.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:21.934 INFO:tasks.workunit.client.0.smithi138.stderr:listed 37 entries 2023-10-14T00:36:22.007 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/296162553 2023-10-14T00:36:22.007 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:36:22.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:22.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:22.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:22.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:22.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:22.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:22.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:22.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:22.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:22.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:24.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:24.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:24.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:24.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:24.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:24.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:24.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:24.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:24.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:24.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:24.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:24.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:25.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:25.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:25.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:25.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:25.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:25.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:25.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:25.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:25.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:25.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:25.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:25.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:25.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:26.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:26.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:26.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:26.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:26.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:26.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:26.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:26.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:26.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:28.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:28.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:28.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:28.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:28.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:28.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:28.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:28.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:28.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:28.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:30.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:30.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:30.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:30.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:30.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:30.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:30.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:30.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:30.382 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:30.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:30.393 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:30.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:30.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:30.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:30.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:30.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:30.427 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:30.429+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:31.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:31.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:31.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:31.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:31.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:31.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:31.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:36:31.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:31.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:31.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:31.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:31.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:31.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:32.028 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:36:32.028 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:32.033 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:32.033 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:33.131 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2550805914 2023-10-14T00:36:33.137 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:36:33.138 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243793 2023-10-14T00:36:33.139 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:33.139 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2550805914 ]] 2023-10-14T00:36:33.139 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2550805914 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\9\6\1\6\2\5\5\3 ]] 2023-10-14T00:36:33.139 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2550805914 2023-10-14T00:36:33.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:33.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:33.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:33.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:34.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:34.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:34.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:34.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:34.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:34.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:34.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:34.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:34.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:34.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:34.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:34.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:35.634 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2550805914 until 2023-10-14T01:36:34.933826+0000 (3600 sec) 2023-10-14T00:36:35.681 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2550805914 2023-10-14T00:36:35.682 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:36:35.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:35.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:35.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:35.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:35.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:35.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:35.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:35.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:35.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:35.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:35.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:35.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:36.778 INFO:tasks.workunit.client.0.smithi138.stderr:listed 38 entries 2023-10-14T00:36:36.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2550805914 2023-10-14T00:36:36.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:36:37.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:37.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:37.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:37.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:37.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:37.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:37.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:37.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:37.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:37.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:37.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:37.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:37.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:37.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:37.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:37.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:37.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:38.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:38.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:36:38.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:38.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:38.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:38.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:38.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:38.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:38.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:38.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:38.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:38.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:38.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:38.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:39.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:39.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:39.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:39.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:39.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:39.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:39.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:39.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:40.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:40.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:40.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:40.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:40.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:40.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:40.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:40.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:40.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:40.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:40.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:40.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:45.096 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:45.096+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:45.118 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:45.116+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:45.121 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:45.120+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:45.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:45.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:45.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:45.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:45.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:45.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:45.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:45.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:45.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:45.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:46.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:46.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:36:46.826 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:46.827 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:36:46.829 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:46.837 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:46.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:46.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:46.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:46.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:46.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:46.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:46.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:46.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:46.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:46.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:46.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:46.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:48.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:48.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:48.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:48.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:48.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:48.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:48.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:48.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:48.376 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2550805914 2023-10-14T00:36:48.381 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:36:48.383 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243808 2023-10-14T00:36:48.383 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:48.383 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2550805914 ]] 2023-10-14T00:36:48.383 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2550805914 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\0\8\0\5\9\1\4 ]] 2023-10-14T00:36:48.384 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:36:49.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:49.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:49.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:49.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:49.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:49.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:49.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:49.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:49.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:49.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:49.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:50.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:50.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:50.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:50.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:50.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:50.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:50.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:50.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:50.246 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:50.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:50.528 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:36:50.528+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:36:50.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:50.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:50.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:50.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:50.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:50.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:50.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:50.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:50.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:50.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:52.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:52.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:52.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:52.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:52.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:52.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:52.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:52.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:52.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:52.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:52.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:52.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:53.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:53.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:53.871 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:53.868+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:53.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:53.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:53.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:53.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:53.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:53.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:53.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:53.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:55.267 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:55.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:36:55.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:55.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:36:55.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:55.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:36:55.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:55.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:36:55.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:55.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:36:55.309 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:36:55.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:36:57.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:57.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:57.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:57.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:57.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:57.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:57.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:57.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:57.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:57.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:58.386 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:58.387 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:36:58.389 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:58.401 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:58.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:58.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:58.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:58.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:58.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:58.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:58.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:58.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:58.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:58.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:59.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:59.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:36:59.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:59.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:36:59.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:59.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:36:59.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:59.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:36:59.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:59.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:36:59.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:59.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:36:59.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:36:59.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:36:59.730 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2550805914 2023-10-14T00:36:59.730 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:36:59.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243819 2023-10-14T00:36:59.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:59.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2550805914 ]] 2023-10-14T00:36:59.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2550805914 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\0\8\0\5\9\1\4 ]] 2023-10-14T00:36:59.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:37:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.008+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.008+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9358600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.012+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.012+0000 7f082630d640 -1 librbd::ImageState: 0x5627d928db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.017 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.016+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.017 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.016+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:00.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:00.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.032+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.032+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8debf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.036+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.036+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.036+0000 7f082630d640 -1 librbd::ImageState: 0x5627d905c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.036+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:00.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.068+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.068+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9382b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.072+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.072+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9382880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.072+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.072+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9460780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.072+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.072+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9460400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.084+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.085 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.084+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9461b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.100+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.102 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.100+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8dd9580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.112+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.112+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9460800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.128+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.128+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9461e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.128+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.130 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.128+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9512000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.143 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.144+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.144 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.144+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9512200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.160 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.160+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.161 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.160+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9512400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.170 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.168+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.171 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.168+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.171 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.168+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9512600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.171 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.168+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9512800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.174 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.172+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.174 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:37:00.172+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9512a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:37:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:37:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:00.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:00.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:00.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:00.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:00.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:00.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:01.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:01.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:01.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:01.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:01.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:01.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:01.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:01.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:01.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:01.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:01.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:01.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:02.591 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:02.584+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:02.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:02.600+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:02.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:02.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:02.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:02.660+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:02.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:02.660+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:02.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:02.660+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:03.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:03.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:03.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:03.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:03.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:03.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:03.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:03.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:04.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:04.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:04.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:04.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:04.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:04.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:04.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:04.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:04.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:04.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:04.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:04.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:04.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:04.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:05.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:05.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:05.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:05.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:05.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:05.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:05.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:05.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:05.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:05.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:05.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:05.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:05.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:05.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:07.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:07.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:07.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:07.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:07.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:07.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:07.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:07.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:08.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:08.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:08.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:08.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:08.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:08.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:08.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:08.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:08.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:08.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:09.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:09.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:09.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:09.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:09.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:09.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:09.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:09.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:09.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:09.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:09.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:09.750 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:09.750 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:09.760 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:37:09.784 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:10.999 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2550805914 2023-10-14T00:37:11.004 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:37:11.017 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243831 2023-10-14T00:37:11.017 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:11.017 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2550805914 ]] 2023-10-14T00:37:11.017 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2550805914 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\0\8\0\5\9\1\4 ]] 2023-10-14T00:37:11.017 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:37:12.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:12.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:12.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:12.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:12.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:12.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:12.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:12.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:12.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:12.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:12.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:12.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:13.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:13.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:13.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:13.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:13.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:13.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:13.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:13.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:13.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:13.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:13.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:13.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:14.102 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:37:14.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.112+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:37:14.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.112+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:14.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:14.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:14.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.136+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:14.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:14.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:14.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:14.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:14.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:14.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:14.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:14.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:14.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:14.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:14.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:15.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:15.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:15.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:15.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:15.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:15.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:15.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:15.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:15.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:15.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:15.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:15.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:17.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:37:17.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:17.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:17.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:17.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:17.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:17.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:17.235 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:17.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:18.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:18.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:18.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:18.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:18.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:18.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:18.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:18.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:18.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:18.227 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:18.223+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:37:18.227 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:18.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:18.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:18.263+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:18.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:18.267+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:18.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:18.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:18.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:18.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:18.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:18.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:19.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:19.031+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:37:19.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:19.031+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:19.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:19.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:19.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:19.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:19.055 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:19.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:19.055 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:19.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:19.055 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:19.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:19.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:19.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:19.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:19.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:19.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:19.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:19.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:19.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:20.152 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:37:20.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:20.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:37:20.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:20.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:20.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:20.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:20.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:20.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:20.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:20.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:20.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:20.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:20.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:20.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:20.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:20.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:20.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:20.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:20.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:20.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:20.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:20.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:20.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:21.027 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:21.028 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:21.033 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:37:21.053 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:22.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:22.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:22.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:22.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:22.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:22.439 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2550805914 2023-10-14T00:37:22.455 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:37:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243842 2023-10-14T00:37:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2550805914 ]] 2023-10-14T00:37:22.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2550805914 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\0\8\0\5\9\1\4 ]] 2023-10-14T00:37:22.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:37:22.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:22.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:22.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:22.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:22.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:22.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:22.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:22.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:23.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:23.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:23.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:23.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:23.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:23.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:23.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:23.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:23.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:23.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:23.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:23.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:25.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:25.599+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:26.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:26.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:26.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:26.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:26.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:26.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:26.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:26.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:26.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:26.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:26.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:26.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:29.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:29.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:29.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:29.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:29.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:29.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:29.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:29.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:30.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:37:30.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:30.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:30.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:30.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:30.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:30.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:30.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:30.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:30.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:30.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:30.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:30.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:30.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:30.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:30.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:32.041 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:32.039+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:32.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:32.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:32.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:32.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:32.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:32.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:32.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:32.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:32.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:32.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:32.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:32.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:32.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:32.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:32.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:32.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:32.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:32.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:32.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:32.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:32.463 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:32.465 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:37:32.488 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:33.854 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2550805914 2023-10-14T00:37:33.877 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:37:33.895 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243853 2023-10-14T00:37:33.896 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:33.896 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2550805914 ]] 2023-10-14T00:37:33.896 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2550805914 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\0\8\0\5\9\1\4 ]] 2023-10-14T00:37:33.896 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:37:34.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:34.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:37:34.362 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:34.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:34.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:34.395+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:34.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:34.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:34.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:34.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:34.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:34.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:34.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:34.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:34.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:34.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:34.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:34.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:34.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:34.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:36.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:36.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:36.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:36.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:38.578 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:38.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:38.578 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:38.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:38.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:38.579+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:38.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:38.587+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:38.624 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:38.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:38.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:38.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:39.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:39.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:39.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:39.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:39.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:39.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:39.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:39.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:39.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:39.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:39.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:39.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:39.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:39.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:39.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:39.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:40.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:40.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:40.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:40.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:40.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:40.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:40.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:40.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:40.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:41.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:41.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:41.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:41.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:41.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:41.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:41.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:41.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:41.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:41.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:41.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:41.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:41.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:41.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:41.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:41.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:41.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:41.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:41.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:41.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:41.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:42.388 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:37:42.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.395+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:37:42.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.395+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:42.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.411+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:42.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.415+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:42.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:42.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:42.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:42.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:42.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:42.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:42.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:42.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:42.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:42.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:42.955+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:42.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:42.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:42.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:42.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:42.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:43.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:43.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:43.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:43.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:43.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:43.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:43.897 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:37:43.897 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:43.898 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:43.900 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:45.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3271336887 2023-10-14T00:37:45.256 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:37:45.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243865 2023-10-14T00:37:45.257 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:45.257 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3271336887 ]] 2023-10-14T00:37:45.257 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3271336887 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\0\8\0\5\9\1\4 ]] 2023-10-14T00:37:45.257 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3271336887 2023-10-14T00:37:45.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:45.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:45.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:45.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:45.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:45.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:45.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:45.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:37:45.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:45.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:45.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:45.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:45.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:45.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:45.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:45.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:45.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:46.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:46.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:46.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:46.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3271336887 until 2023-10-14T01:37:46.987748+0000 (3600 sec) 2023-10-14T00:37:47.727 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3271336887 2023-10-14T00:37:47.727 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:37:48.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:48.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:48.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:48.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:48.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:48.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:48.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:48.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:48.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:48.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:48.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:48.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:48.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:48.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:49.019 INFO:tasks.workunit.client.0.smithi138.stderr:listed 39 entries 2023-10-14T00:37:49.069 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3271336887 2023-10-14T00:37:49.070 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:37:50.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:50.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:50.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:50.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:50.181 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:50.179+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:50.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:50.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:50.196 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:50.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:50.196 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:50.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:50.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:50.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:50.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:50.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:50.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:50.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:50.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:50.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:50.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:50.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:50.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:52.239 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:37:52.240 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:52.240 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:37:52.246 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:52.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.251+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:37:52.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.251+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:37:52.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.251+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:37:52.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.399+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:52.421 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:52.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:52.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:37:52.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:37:52.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:37:52.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:52.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:52.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:52.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:52.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:52.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:52.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:52.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:52.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:52.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:52.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:52.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:53.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:53.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:53.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:53.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:53.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:53.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:53.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:53.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:53.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:53.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:53.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:53.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:53.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:53.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:55.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:55.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:55.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:55.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:55.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:55.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:55.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:55.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:55.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:55.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:55.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:56.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:56.363+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:56.374 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:56.371+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:56.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:56.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:56.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:56.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:56.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:56.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:56.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:56.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:37:56.495 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:37:56.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:37:56.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:56.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:56.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:56.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:56.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:56.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:57.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:57.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:57.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:57.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:57.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:57.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:57.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:57.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:57.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:57.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:57.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:57.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:58.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:58.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:58.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:58.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:58.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:58.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:58.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:58.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:58.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:58.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:59.088 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:37:59.089 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:59.089 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:59.089 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:59.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:59.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:59.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:59.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:37:59.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:37:59.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:37:59.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:37:59.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:37:59.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:37:59.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:37:59.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:37:59.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:00.006 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.002+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.007 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.002+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9461200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.050+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.050+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd8a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.050+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.050+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.052 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.050+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d928c480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.050+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd8600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.066+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d928db80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.066+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d905f180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.066+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8860080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.093 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.090+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.090+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9359e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.111 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f84400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.124 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.122+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.124 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.122+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9382600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.124 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.122+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.125 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.122+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9108180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.195 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.190+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.195 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.190+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d945ff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.196 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.196 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.196 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.196 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.197 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.197 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.197 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9512400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.197 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9512c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.197 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.198 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.198 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.198 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.194+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.224 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.222+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.222+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.222+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.225 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:38:00.222+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:38:00.789 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3271336887 2023-10-14T00:38:00.790 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:38:00.790 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243880 2023-10-14T00:38:00.790 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:00.790 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3271336887 ]] 2023-10-14T00:38:00.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3271336887 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\2\7\1\3\3\6\8\8\7 ]] 2023-10-14T00:38:00.791 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:38:01.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:01.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:01.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:01.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:01.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:01.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:02.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:02.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:02.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:02.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:02.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:02.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:03.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:03.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:03.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:03.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:03.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:03.463 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:03.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:03.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:03.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:03.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:03.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:03.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:03.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:03.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:03.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:03.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:03.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:03.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:03.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:03.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:03.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:03.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:03.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:03.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:03.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:04.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:04.014+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:04.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:04.014+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:04.038 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:04.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:04.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:04.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:04.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:04.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:04.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:04.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:06.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:06.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:06.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:06.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:06.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:06.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:06.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:06.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:07.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:07.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:07.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:07.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:07.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:07.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:07.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:07.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:08.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:08.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:08.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:08.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:08.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:08.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:08.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:08.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:08.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:08.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:08.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:10.795 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:38:10.796 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:10.796 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:10.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:10.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:10.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:10.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:10.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:10.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:10.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:10.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:10.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:11.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:10.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:11.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:11.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:11.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:11.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:11.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:11.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:11.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:11.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:11.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:11.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:11.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:11.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:11.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:11.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:12.274 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3271336887 2023-10-14T00:38:12.277 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:38:12.285 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243892 2023-10-14T00:38:12.285 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:12.286 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3271336887 ]] 2023-10-14T00:38:12.286 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3271336887 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\2\7\1\3\3\6\8\8\7 ]] 2023-10-14T00:38:12.286 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:38:12.362 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:12.358+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:12.363 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:12.358+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:12.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:12.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:12.411 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:12.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:12.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:12.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:12.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:12.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:12.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:12.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:13.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:13.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:13.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:13.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:13.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:13.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:13.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:13.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:14.093 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:38:14.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:14.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:14.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.138+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:14.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.154+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:14.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.166+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:14.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.166+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:14.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.166+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:14.495 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:38:14.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:14.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:14.503 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:14.503 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:14.503 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:14.503 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:14.503 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:14.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:14.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:15.235 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:15.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:15.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:15.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:15.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:15.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:15.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:15.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:15.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:15.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:15.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:15.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:15.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:15.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:15.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:15.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:15.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:15.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:16.411 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:16.406+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:16.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:16.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:16.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:16.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:16.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:16.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:16.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:16.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:16.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:16.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:16.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:16.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:16.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:16.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:18.096 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:18.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:18.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:18.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:18.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:18.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:18.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:18.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:18.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:18.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:19.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:19.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:19.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:19.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:19.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:19.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:19.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:19.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:19.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:19.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:19.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:19.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:19.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:19.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:20.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:20.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:20.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:20.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:20.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:20.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:20.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:20.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:20.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:20.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:38:22.323 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:22.347 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:22.350 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:23.624 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3271336887 2023-10-14T00:38:23.625 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:38:23.625 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243903 2023-10-14T00:38:23.625 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:23.626 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3271336887 ]] 2023-10-14T00:38:23.626 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3271336887 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\2\7\1\3\3\6\8\8\7 ]] 2023-10-14T00:38:23.626 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:38:23.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:23.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:23.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:23.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:23.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:23.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:23.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:23.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:23.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:24.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:24.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:24.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:24.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:24.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:24.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:24.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:24.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:24.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:25.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:25.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:25.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:25.950+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:26.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:26.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:26.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:26.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:26.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:26.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:27.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:27.646+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:27.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:27.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:27.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:27.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:27.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:27.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:27.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:27.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:27.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:27.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:27.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:27.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:27.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:27.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:27.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:27.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:27.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:27.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:27.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:27.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:27.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:27.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:27.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:27.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:28.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:38:28.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:38:28.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:28.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:28.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:28.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:28.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:28.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:28.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:28.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:28.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:30.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:30.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:30.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:30.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:31.467 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:31.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:31.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:31.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:32.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:32.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:32.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:32.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:32.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:32.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:32.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:32.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:32.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:32.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:32.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:32.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:32.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:32.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:33.645 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:38:33.645 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:33.648 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:33.648 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:33.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:33.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:33.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:33.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:33.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:33.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:33.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:33.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:33.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:33.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:33.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:33.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:33.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:33.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:34.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:34.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:34.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:34.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:34.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:34.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:34.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:34.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:34.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:34.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:34.687 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3271336887 2023-10-14T00:38:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:38:34.689 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243914 2023-10-14T00:38:34.689 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:34.689 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3271336887 ]] 2023-10-14T00:38:34.689 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3271336887 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\2\7\1\3\3\6\8\8\7 ]] 2023-10-14T00:38:34.690 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:38:37.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:37.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:37.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:37.200 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:37.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:37.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:37.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:37.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:37.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:37.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:37.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:37.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:38.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:38.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:38.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:38.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:38.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:38.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:40.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:40.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:40.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:40.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:40.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:40.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:40.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:40.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:40.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:40.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:40.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:40.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:41.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:41.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:41.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:41.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:41.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:41.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:41.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:41.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:41.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:41.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:41.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:41.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:41.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:41.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:41.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:41.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:41.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:41.389+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:41.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:41.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:41.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:41.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:41.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:41.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:42.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:42.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:42.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:42.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:42.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:42.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:42.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:42.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:42.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:42.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:42.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:42.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:42.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:42.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:42.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:42.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:42.460 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:42.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:42.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:42.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:42.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:42.477+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:42.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:42.477+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:42.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:42.477+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:42.480 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:42.477+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:43.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:43.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:43.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:43.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:43.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:43.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:43.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:43.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:43.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:43.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:44.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:44.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:44.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:44.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:44.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:44.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:44.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:44.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:44.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:44.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:44.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:44.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:44.717 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:38:44.717 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:44.718 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:44.719 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:45.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:45.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:45.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:45.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:45.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:45.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:45.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:45.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:45.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:45.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:46.182 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3271336887 2023-10-14T00:38:46.188 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:38:46.189 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243926 2023-10-14T00:38:46.189 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:46.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3271336887 ]] 2023-10-14T00:38:46.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3271336887 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\2\7\1\3\3\6\8\8\7 ]] 2023-10-14T00:38:46.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:38:46.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:46.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:46.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:46.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:46.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:46.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:46.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:46.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:47.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:47.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:47.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:47.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:47.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:47.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:47.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:47.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:47.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:47.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:47.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:47.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:48.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:48.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:48.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:48.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:48.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:48.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:48.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:48.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:48.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:48.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:48.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:48.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:48.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:48.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:48.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:48.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:48.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:48.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:48.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:48.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:48.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.777+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:48.788 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:48.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.789+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:48.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:48.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:48.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:49.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:49.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:49.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:49.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:49.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:49.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:49.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:49.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:49.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:49.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:49.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:49.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:49.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:49.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:51.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:51.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:51.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:51.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:52.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:52.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:38:52.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:52.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:38:52.121 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:52.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:38:52.121 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:52.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:52.124 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:52.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:52.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:52.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:52.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:52.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:52.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:52.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:52.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:52.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:52.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:52.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:55.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:55.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:38:55.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:55.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:38:55.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:55.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:38:55.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:38:55.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:38:56.191 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:56.194 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:56.196 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:38:56.202 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:57.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:38:57.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:57.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:38:57.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:38:57.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:57.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:57.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:57.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:57.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:38:57.735 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3912908247 2023-10-14T00:38:57.735 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:38:57.736 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243937 2023-10-14T00:38:57.736 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:57.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3912908247 ]] 2023-10-14T00:38:57.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3912908247 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\2\7\1\3\3\6\8\8\7 ]] 2023-10-14T00:38:57.737 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/3912908247 2023-10-14T00:38:58.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:58.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:38:58.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:58.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:38:58.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:58.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:38:58.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:58.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:38:58.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:38:58.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:00.005 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.005+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.006 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.005+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8ae2380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.013 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.013+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.013+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8e17d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8b18900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8e17600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9382580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9358380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9513a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.029+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9513900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.037+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.037+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.049+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.049+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.061+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.061+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9383c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.061+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.061+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9513b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.061+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.065+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.073+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.073+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.073+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.073 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.073+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.077+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.077+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.077+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.077+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.081+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.081 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.081+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.087 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.085+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.087 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.085+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8a68080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.092 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.093+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.092 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:39:00.093+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8a68280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:39:00.139 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/3912908247 until 2023-10-14T01:38:59.468159+0000 (3600 sec) 2023-10-14T00:39:00.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:39:00.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:39:00.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:00.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:00.189 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:39:00.190 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/3912908247 2023-10-14T00:39:00.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:00.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:39:00.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:39:00.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:39:00.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:00.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:00.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:00.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:00.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:00.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:00.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:00.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:01.384 INFO:tasks.workunit.client.0.smithi138.stderr:listed 40 entries 2023-10-14T00:39:01.420 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/3912908247 2023-10-14T00:39:01.420 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:39:02.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:02.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:39:02.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:02.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:02.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:02.309+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:02.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:02.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:02.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:02.329+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:02.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:02.329+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:02.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:02.329+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:03.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:03.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:03.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:03.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:03.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:03.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:03.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:03.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:03.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:03.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:04.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:04.477+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:04.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:04.493+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:04.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:04.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:04.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:04.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:04.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:04.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:04.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:04.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:05.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:05.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:05.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:05.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:05.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:05.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:05.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:05.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:05.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:05.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:05.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:05.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:05.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:05.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:05.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:05.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:05.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:05.465+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:05.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:05.505+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:05.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:05.529+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:05.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:05.529+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:05.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:05.529+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:06.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:39:06.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:06.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:06.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:06.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:06.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:06.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:06.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:06.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:07.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:39:07.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:07.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:07.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:07.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:07.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:07.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:07.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:07.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:11.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:11.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:11.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:11.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:11.439 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:39:11.440 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:11.442 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:11.452 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:11.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:11.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:11.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:11.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:11.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:11.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:11.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:11.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:11.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:11.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:12.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:12.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:12.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:12.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:12.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:12.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:12.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:12.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:12.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:12.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:12.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:12.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:12.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:12.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:12.843 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3912908247 2023-10-14T00:39:12.845 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:39:12.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243952 2023-10-14T00:39:12.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:12.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3912908247 ]] 2023-10-14T00:39:12.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3912908247 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\1\2\9\0\8\2\4\7 ]] 2023-10-14T00:39:12.852 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:39:13.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:13.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:13.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:13.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:13.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:13.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:13.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:13.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:13.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:13.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:13.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:13.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:13.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:14.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:14.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:14.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:14.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:14.168 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:14.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:14.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:14.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:14.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:14.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:14.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:14.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:14.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:14.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:14.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:14.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:14.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:14.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:18.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:18.345+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:39:18.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:18.345+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:18.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:18.353+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:18.357 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:18.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:18.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:18.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:18.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:18.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:18.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:18.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:18.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:18.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:18.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:18.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:19.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:39:19.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:19.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:19.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:19.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:19.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:19.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:19.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:19.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:20.547 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.544+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:39:20.547 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.544+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:20.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.560+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:20.584 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.584+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:20.666 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:20.666 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:20.667 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:20.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:39:20.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:20.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:20.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.952+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:20.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:20.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:20.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:20.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:22.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:22.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:22.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:22.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:22.873 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:22.873 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:39:22.873 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:22.884 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:23.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:23.136+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:39:23.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:23.136+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:23.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:23.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:23.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:23.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:23.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:23.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:23.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:23.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:23.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:23.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:23.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:23.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:23.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:23.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:23.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:23.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:23.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:23.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:23.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:23.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:23.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:23.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:24.180 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3912908247 2023-10-14T00:39:24.182 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:39:24.183 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243964 2023-10-14T00:39:24.183 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:24.183 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3912908247 ]] 2023-10-14T00:39:24.184 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3912908247 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\1\2\9\0\8\2\4\7 ]] 2023-10-14T00:39:24.184 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:39:25.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:25.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:25.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:25.264+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:26.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:26.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:26.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:26.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:26.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:26.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:26.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:26.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:26.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:26.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:26.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:26.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:26.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:26.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:27.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:27.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:27.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:27.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:27.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:27.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:27.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:27.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:27.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:27.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:27.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:28.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:28.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:28.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:28.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:28.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:28.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:39:28.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:28.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:28.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:28.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:28.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:28.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:28.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:28.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:28.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:28.680+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:28.703 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:28.700+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:28.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:28.712+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:28.712 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:28.712+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:28.712 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:28.712+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:29.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:39:29.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:39:29.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:39:29.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:29.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:29.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:29.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:29.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:29.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:29.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:29.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:31.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:31.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:31.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:31.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:31.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:31.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:31.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:31.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:31.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:31.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:31.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:31.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:31.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:31.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:33.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:33.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:34.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:34.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:34.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:34.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:34.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:34.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:34.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:34.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:34.185 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:34.186 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:39:34.186 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:34.194 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:35.530 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3912908247 2023-10-14T00:39:35.535 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:39:35.550 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243975 2023-10-14T00:39:35.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:35.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3912908247 ]] 2023-10-14T00:39:35.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3912908247 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\1\2\9\0\8\2\4\7 ]] 2023-10-14T00:39:35.551 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:39:35.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:35.708+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:35.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:35.720+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:35.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:35.740+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:35.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:35.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:35.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:35.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:35.780 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:35.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:37.338 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:39:37.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:39:37.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:37.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.356+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:37.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:37.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:37.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:37.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:37.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:38.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:38.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:38.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:38.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:38.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:38.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:38.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:38.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:38.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:38.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:39.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:39.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:39.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:39.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:39.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:39.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:39.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:39.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:39.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:39.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:39.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:39.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:40.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:40.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:40.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:40.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:40.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:40.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:40.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:40.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:40.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:40.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:41.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:41.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:41.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:41.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:41.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:41.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:41.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:41.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:41.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:41.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:41.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:41.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:42.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:42.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:39:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:42.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:42.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:42.984+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:43.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:43.045 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.044+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:43.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.044+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:43.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.044+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:43.843 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:43.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:43.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:43.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:43.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:43.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:44.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:44.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:44.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:44.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:44.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:44.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:44.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:44.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:45.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:45.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:45.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:45.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:45.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:45.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:45.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:45.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:45.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:45.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:45.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:45.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:45.555 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:45.557 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:39:45.558 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:45.564 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:46.854 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3912908247 2023-10-14T00:39:46.854 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:39:46.862 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243986 2023-10-14T00:39:46.863 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:46.863 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3912908247 ]] 2023-10-14T00:39:46.863 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3912908247 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\1\2\9\0\8\2\4\7 ]] 2023-10-14T00:39:46.863 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:39:47.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:47.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:47.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:47.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:47.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:47.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:47.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:47.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:48.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:47.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:48.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:47.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:49.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:49.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:49.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:49.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:49.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:49.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:50.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:50.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:50.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:50.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:50.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:50.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:50.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:50.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:52.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:52.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:52.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:52.716+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:52.726 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:52.724+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:39:52.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:52.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:52.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:52.752+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:52.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:52.752+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:52.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:52.752+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:39:53.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:53.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:53.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:53.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:53.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:53.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:53.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:53.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:53.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:53.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:54.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:54.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:39:54.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:54.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:54.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:54.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:54.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:54.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:54.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:54.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:54.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:54.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:55.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:55.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:55.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:55.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:55.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:55.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:55.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:55.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:39:56.878 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:56.879 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:56.880 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:58.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:58.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:39:58.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:58.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:39:58.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:58.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:39:58.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:58.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:39:58.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:58.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:39:58.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:58.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:39:58.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:39:58.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:39:58.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:58.424+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:39:58.437 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:58.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:39:58.441 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/3912908247 2023-10-14T00:39:58.441 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:39:58.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697243998 2023-10-14T00:39:58.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:58.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/3912908247 ]] 2023-10-14T00:39:58.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/3912908247 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\1\2\9\0\8\2\4\7 ]] 2023-10-14T00:39:58.443 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:39:58.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:58.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:39:58.517 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:58.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:39:58.517 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:39:58.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:00.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:00.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:00.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:00.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:00.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:00.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:00.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:00.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:00.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:00.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:00.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:00.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:00.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:00.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:02.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:02.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:02.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:02.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:02.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:02.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:04.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:04.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:04.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:04.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:04.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:04.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:04.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:04.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:04.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:06.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:06.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:06.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:06.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:06.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:06.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:06.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:06.471+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:06.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:06.487+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:06.587 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:06.583+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:07.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:07.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:07.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:07.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:07.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:07.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:07.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:07.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:07.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:07.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:07.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:07.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:08.456 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:40:08.459 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:08.482 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:08.486 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:09.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:09.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:09.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:09.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:09.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:09.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:09.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:09.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:09.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:09.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:09.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:09.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:09.822 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2551040678 2023-10-14T00:40:09.823 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:40:09.823 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244009 2023-10-14T00:40:09.823 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:09.824 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2551040678 ]] 2023-10-14T00:40:09.824 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2551040678 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\1\2\9\0\8\2\4\7 ]] 2023-10-14T00:40:09.824 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/2551040678 2023-10-14T00:40:10.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:40:10.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:40:10.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:10.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:10.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:10.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:10.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:10.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:10.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:10.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:10.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:10.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:10.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:10.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:10.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:10.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:10.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:11.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:11.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:11.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:11.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:11.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:11.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:11.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:11.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:11.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:11.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:11.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:11.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:11.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:11.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:11.758 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/2551040678 until 2023-10-14T01:40:11.027322+0000 (3600 sec) 2023-10-14T00:40:11.818 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:40:11.818 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/2551040678 2023-10-14T00:40:12.871 INFO:tasks.workunit.client.0.smithi138.stderr:listed 41 entries 2023-10-14T00:40:12.959 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/2551040678 2023-10-14T00:40:12.959 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:40:16.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:16.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:16.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:16.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:16.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:16.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:16.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:16.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:16.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:16.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:17.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:17.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:17.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:17.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:17.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:17.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:17.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:17.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:17.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:17.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:17.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:17.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:17.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:17.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:17.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:17.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:17.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:17.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:17.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:17.879+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:17.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:17.895+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:19.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:19.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:19.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:19.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:19.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:19.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:19.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:19.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:19.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:19.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:19.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:40:19.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:19.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:19.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:19.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:19.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:19.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:19.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:19.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:20.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:20.415+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:40:20.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:20.415+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:20.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:20.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:20.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:20.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:20.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:20.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:20.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:20.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:21.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:21.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:21.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:21.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:21.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:21.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:21.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:21.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:21.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:22.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:22.903+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:40:22.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:22.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:22.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:22.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:22.972 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:22.973 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:22.974 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:23.020 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:40:23.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:23.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:23.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:23.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:23.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:23.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:23.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:23.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:23.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:24.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:24.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:24.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:24.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:24.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:24.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:24.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:24.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:24.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:24.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:24.675 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2551040678 2023-10-14T00:40:24.675 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:40:24.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244024 2023-10-14T00:40:24.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:24.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2551040678 ]] 2023-10-14T00:40:24.676 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2551040678 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\1\0\4\0\6\7\8 ]] 2023-10-14T00:40:24.677 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:40:24.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:24.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:24.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:24.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:25.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:25.175+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:25.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:25.211+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:25.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:25.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:25.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:25.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:25.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:25.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:26.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:26.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:26.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:26.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:26.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:26.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:26.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:26.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:26.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:26.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:27.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:27.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:27.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:27.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:27.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:27.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:27.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:27.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:28.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:28.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:28.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:28.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:28.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:28.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:28.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:28.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:28.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:28.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:28.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:28.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:28.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:28.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:28.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:28.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:28.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:28.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:28.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:28.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:28.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:28.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:28.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:28.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:29.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:29.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:29.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:29.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:29.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:29.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:29.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:29.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:29.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:29.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:30.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:30.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:30.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:30.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:30.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:30.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:30.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:30.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:30.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:30.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:30.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:30.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:30.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:30.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:31.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:31.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:31.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:31.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:31.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:31.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:31.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:31.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:31.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:31.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:31.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:31.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:32.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:32.235+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:32.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:32.255+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:32.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:32.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:32.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:32.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:32.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:32.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:33.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:33.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:33.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:33.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:33.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:33.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:33.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:33.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:33.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:33.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:34.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:34.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:34.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:34.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:34.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:34.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:34.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:34.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:34.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:34.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:34.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:34.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:34.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:34.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:34.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:34.584 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:34.583+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:40:34.584 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:34.583+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:34.593 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:34.587+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:34.593 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:34.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:34.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:34.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:34.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:34.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:34.678 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:40:34.685 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:34.699 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:34.704 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:35.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:40:35.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:35.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:35.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:35.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:35.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:35.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:35.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:35.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:36.022 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2551040678 2023-10-14T00:40:36.022 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:40:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244036 2023-10-14T00:40:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2551040678 ]] 2023-10-14T00:40:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2551040678 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\1\0\4\0\6\7\8 ]] 2023-10-14T00:40:36.024 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:40:36.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:36.031+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:36.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:36.039+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:36.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:36.043+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:36.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:36.043+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:36.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:36.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:36.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:36.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:36.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:36.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:36.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:36.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:36.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:36.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:37.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:37.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:37.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:37.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:37.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:37.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:37.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:37.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:37.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:37.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:37.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:38.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:38.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:38.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:38.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:38.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:38.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:38.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:38.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:38.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:38.410 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:38.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:38.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:38.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:38.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:38.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:38.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:38.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:38.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:38.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:38.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:38.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:38.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:38.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:38.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:38.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:39.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:39.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:39.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:39.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:39.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:39.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:39.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:39.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:39.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:39.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:39.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:41.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:40:41.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:41.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:41.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:41.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:41.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:41.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:41.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:40:41.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:41.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:41.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:41.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:41.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:41.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:41.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:41.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:41.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:41.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:41.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:41.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:41.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:41.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:41.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:41.711+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:40:41.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:41.711+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:41.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:41.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:41.733 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:41.731+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:42.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:42.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:42.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:42.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:42.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:42.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:42.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:42.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:42.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:42.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:42.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:42.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:42.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:42.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:43.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:43.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:43.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:43.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:43.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:43.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:43.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:43.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:43.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:43.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:43.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:43.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:43.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:43.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:40:43.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:43.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:43.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:43.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:43.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:43.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:43.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:43.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:43.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:43.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:46.026 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:40:46.026 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:46.033 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:46.044 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:46.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:46.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:46.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:46.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:46.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:46.342+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:46.350 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:46.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:46.350 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:46.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:46.350 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:46.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:47.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:47.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:40:47.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:47.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:47.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:47.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:47.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:47.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:47.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:47.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:47.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:47.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:47.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:47.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:47.340 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2551040678 2023-10-14T00:40:47.340 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:40:47.340 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244047 2023-10-14T00:40:47.340 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2551040678 ]] 2023-10-14T00:40:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2551040678 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\1\0\4\0\6\7\8 ]] 2023-10-14T00:40:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:40:47.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:47.342+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:47.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:47.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:47.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:47.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:47.355 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:47.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:47.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:47.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:47.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:47.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:49.618 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:49.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:49.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:49.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:49.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:49.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:49.666 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:49.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:49.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:49.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:50.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:50.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:50.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.526+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:50.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:50.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:50.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:50.578 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:50.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:50.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:50.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:50.594+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:51.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:51.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:51.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:51.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:51.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:51.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:51.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:52.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:51.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:52.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:52.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:52.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:52.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:52.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:52.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:52.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:52.930+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:52.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:52.930+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:52.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:52.930+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:54.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:54.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:54.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:54.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:54.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:54.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:54.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:54.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:54.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:54.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:56.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:56.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:56.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:56.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:56.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:56.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:56.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:56.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:56.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:56.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:56.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:56.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:56.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:56.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:57.342 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:40:57.343 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:57.343 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:57.364 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:57.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:57.846+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:40:57.877 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:57.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:57.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:57.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:57.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:57.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:57.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:57.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:57.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:57.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:57.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:57.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:57.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:57.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:58.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:57.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:58.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:58.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:58.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:58.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:58.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:58.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:58.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:58.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:40:58.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:58.582+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:40:58.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:40:58.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:40:58.608 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:58.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:40:58.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:40:58.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:40:58.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:40:58.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:58.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:40:58.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:58.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:40:58.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:40:58.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:40:58.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:40:58.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:40:58.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:40:58.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:40:58.751 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2551040678 2023-10-14T00:40:58.755 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:40:58.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244058 2023-10-14T00:40:58.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:58.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2551040678 ]] 2023-10-14T00:40:58.756 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2551040678 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\1\0\4\0\6\7\8 ]] 2023-10-14T00:40:58.757 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:41:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.026+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.032 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.026+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9487080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.030+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.030+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.038 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.030+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.030+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd8000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.030+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9512b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.039 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.030+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8dd8780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.038+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.042 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.038+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9512f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.050+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.055 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.050+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.058+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.058+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.058+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.063 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.058+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9513f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.069 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.070 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9460a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9461400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8ae3480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8f85180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.066+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8f85c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.070+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.070+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9460e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.082 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.078+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.082 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.078+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9359a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.087 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.082+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.087 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.082+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9382100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.086+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.091 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.086+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d9382300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.102+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.102+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8a68900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.106+0000 7f0826b0e640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.110 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:41:00.106+0000 7f0826b0e640 -1 librbd::ImageState: 0x5627d8a68380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:41:00.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:00.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:00.637 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:00.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:00.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:00.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:00.687 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:00.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:00.687 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:00.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:00.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:00.678+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:00.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:00.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:00.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:00.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:00.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:00.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:00.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:00.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:00.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:00.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:00.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:00.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:41:01.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:41:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:01.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:01.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:01.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:01.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:01.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:01.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:02.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:02.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:02.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:02.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:02.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:02.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:02.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:03.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:03.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:03.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:03.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:03.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:03.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:03.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:03.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:05.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:05.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:05.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:05.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:05.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:05.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:05.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:05.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:05.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:05.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:05.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:05.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:05.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:05.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:07.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:07.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:07.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:07.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:07.026 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:07.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:07.027 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:07.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:07.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:07.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:07.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:07.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:07.041 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:07.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:08.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:41:08.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:08.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:08.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.618+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:08.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:08.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:08.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:08.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:08.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:08.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:08.761 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:08.761 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:08.766 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:41:08.786 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:09.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:09.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:09.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:09.790+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:09.819 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:09.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:09.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:09.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:09.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:09.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:09.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:09.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:10.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:10.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:10.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:10.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:10.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:10.242 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/2551040678 2023-10-14T00:41:10.243 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:41:10.243 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244070 2023-10-14T00:41:10.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:10.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/2551040678 ]] 2023-10-14T00:41:10.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/2551040678 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\1\0\4\0\6\7\8 ]] 2023-10-14T00:41:10.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:41:10.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:10.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:10.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:10.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:10.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:10.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:10.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:10.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:10.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:10.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:10.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:10.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:10.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:10.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:10.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:10.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:10.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:10.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:11.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:11.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:11.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:11.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:11.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:11.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:11.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:11.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:11.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:11.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:11.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:11.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:11.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:11.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:16.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:16.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:16.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:16.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:16.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:16.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:16.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:16.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:16.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:16.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:16.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:17.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:17.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:17.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:17.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:17.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:17.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:17.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:17.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:17.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:17.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:17.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:17.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:17.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:18.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:18.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:18.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:18.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:18.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:18.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:18.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:18.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:18.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:18.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:18.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:18.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:18.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:18.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:18.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:19.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:19.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:19.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:19.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:19.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:19.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:19.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:19.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:19.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:19.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:19.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:19.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:19.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:19.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:19.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:19.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:19.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:20.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:20.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:20.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:20.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:20.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:20.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:20.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:20.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:20.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:20.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:20.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:20.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:41:20.262 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:20.263 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:21.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:21.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:21.623 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/263115127 2023-10-14T00:41:21.624 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:41:21.630 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244081 2023-10-14T00:41:21.630 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:21.630 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/263115127 ]] 2023-10-14T00:41:21.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/263115127 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\5\5\1\0\4\0\6\7\8 ]] 2023-10-14T00:41:21.631 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/263115127 2023-10-14T00:41:21.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:21.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:21.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:21.638+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:21.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:21.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:21.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:21.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:21.689 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:21.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:21.689 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:21.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:21.689 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:21.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:21.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:21.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:21.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:21.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:21.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:22.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:21.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:22.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:22.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:22.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:22.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:23.782 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/263115127 until 2023-10-14T01:41:23.096893+0000 (3600 sec) 2023-10-14T00:41:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:41:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/263115127 2023-10-14T00:41:24.876 INFO:tasks.workunit.client.0.smithi138.stderr:listed 42 entries 2023-10-14T00:41:24.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:24.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:24.975 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/263115127 2023-10-14T00:41:24.975 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:41:24.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:24.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:24.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:24.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:24.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:24.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:25.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:25.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:25.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:25.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:25.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:25.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:25.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:25.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:25.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:25.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:25.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:25.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:25.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:25.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:25.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:25.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:25.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:25.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:25.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:25.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:25.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:25.966+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:26.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:25.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:26.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:25.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:26.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:25.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:26.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:26.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:26.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:26.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:26.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:26.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:26.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:26.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:26.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:26.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:28.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:28.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:28.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:28.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:28.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:28.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:28.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:28.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:28.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:28.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:28.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:28.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:29.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:29.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:29.251 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:29.245+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:29.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:29.273+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:29.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:29.321+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:29.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:29.321+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:29.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:29.321+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:31.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:31.525+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:31.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:31.545+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:31.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:31.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:31.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:31.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:31.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:31.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:31.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:31.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:31.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:31.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:31.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:31.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:31.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:31.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:32.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:32.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:32.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:32.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:32.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:32.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:32.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:32.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:32.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:32.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:33.734 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:33.729+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:33.734 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:33.729+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:33.755 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:33.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:33.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:33.777+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:33.798 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:33.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:33.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:33.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:33.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:33.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:34.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:34.297+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:34.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:34.297+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:34.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:34.337+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:34.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:34.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:34.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:34.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:34.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:34.469+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:34.989 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:41:34.989 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:34.991 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:34.993 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:35.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:35.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:35.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:35.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:35.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:35.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:35.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:35.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:35.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:35.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:35.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:35.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:35.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:35.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:35.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:35.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:36.094 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/263115127 2023-10-14T00:41:36.095 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:41:36.095 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244096 2023-10-14T00:41:36.095 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:36.095 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/263115127 ]] 2023-10-14T00:41:36.096 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/263115127 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\6\3\1\1\5\1\2\7 ]] 2023-10-14T00:41:36.096 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:41:36.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:36.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:36.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:36.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:36.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:36.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:36.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:36.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:36.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:36.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:36.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:36.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:37.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:37.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:41:37.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:37.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:37.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:37.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:37.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:37.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:37.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:37.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:37.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:37.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:38.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:38.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:38.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:38.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:38.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:38.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:38.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:38.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:38.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:38.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:38.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:38.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:39.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:39.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:39.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:39.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:39.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:39.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:39.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:39.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:39.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:39.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:39.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:39.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:39.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:39.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:40.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:40.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:40.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:40.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:40.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:40.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:40.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:40.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:40.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:40.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:43.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:43.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:43.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:43.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:43.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:43.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:43.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:43.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:43.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:43.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:43.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:43.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:43.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:43.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:43.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:43.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:43.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:43.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:43.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:43.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:43.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:44.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:41:44.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:44.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:44.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:44.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:44.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:44.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:44.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:44.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:44.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:44.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:44.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:44.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:44.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:44.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:44.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:45.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:45.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:45.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:45.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:45.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:45.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:45.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:45.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:45.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:45.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:45.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:45.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:45.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:45.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:45.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:45.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:45.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:45.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:45.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:45.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:45.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:45.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:45.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:45.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:46.102 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:41:46.104 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:46.112 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:46.117 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:46.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:46.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:46.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:46.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:46.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:46.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:46.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:46.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:46.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:47.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:47.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:47.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:47.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:47.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:47.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:47.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:47.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:47.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:47.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:47.198 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/263115127 2023-10-14T00:41:47.198 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:41:47.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244107 2023-10-14T00:41:47.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:47.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/263115127 ]] 2023-10-14T00:41:47.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/263115127 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\6\3\1\1\5\1\2\7 ]] 2023-10-14T00:41:47.200 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:41:49.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:49.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:49.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:49.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:49.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:49.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:49.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:49.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:49.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:49.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:49.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:49.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:49.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:50.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:50.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:50.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:50.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:50.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:50.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:50.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:50.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:50.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:50.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:50.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:50.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:50.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:50.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:51.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:51.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:41:51.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:41:51.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:51.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:51.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:51.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:51.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:51.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:51.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:51.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:51.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:51.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:51.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:51.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:51.876 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:51.877+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:51.876 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:51.877+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:51.895 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:51.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:51.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:51.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:51.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:51.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:51.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:51.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:52.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:52.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:52.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:52.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:52.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:52.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:52.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:52.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:52.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:52.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:52.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:52.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:53.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:53.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:53.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:53.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:53.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:53.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:53.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:53.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:53.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:53.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:54.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:54.157+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:54.160 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:54.157+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:54.182 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:54.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:54.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:54.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:54.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:54.205+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:54.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:54.205+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:55.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:55.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:55.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:55.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:55.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:55.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:55.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:55.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:55.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:55.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:55.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:55.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:55.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:55.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:41:55.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:55.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:55.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:55.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:55.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:55.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:56.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:56.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:41:56.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:56.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:41:56.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:56.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:41:56.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:56.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:41:56.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:56.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:41:56.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:56.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:41:56.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:41:56.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:41:57.210 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:57.212 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:57.212 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:57.213 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:41:58.357 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:58.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:41:58.357 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:58.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:41:58.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:58.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:41:58.411 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:58.409+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:41:58.419 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:58.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:41:58.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:58.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:41:58.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:41:58.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:41:58.656 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/263115127 2023-10-14T00:41:58.657 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:41:58.658 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244118 2023-10-14T00:41:58.658 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:58.659 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/263115127 ]] 2023-10-14T00:41:58.659 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/263115127 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\6\3\1\1\5\1\2\7 ]] 2023-10-14T00:41:58.659 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:42:00.009 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.009+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.009+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d905fa00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.010 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.009+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.009+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.009+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9513400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.009+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.013+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.014 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.013+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9513f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.028 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.025+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.029 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.025+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8ae3200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.029+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.029+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dea900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.049+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.051 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.049+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d896da80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.053 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.053+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.054 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.053+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.057+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.057 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.057+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9383b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.073+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.073+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.073+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9487980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.074 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.073+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945f080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.077 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.077+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.078 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.077+0000 7f082630d640 -1 librbd::ImageState: 0x5627d945eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.084 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.085+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.084 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.085+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8a68600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.089 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.089+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.089 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.089+0000 7f082630d640 -1 librbd::ImageState: 0x5627d8a68a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.097+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.097+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9383380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.106 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.105+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.106 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.105+0000 7f082630d640 -1 librbd::ImageState: 0x5627d9383680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.107 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.105+0000 7f082630d640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.107 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.105+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9109800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.113+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.112 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.113+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d928cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.120 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.117+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.121 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.121+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8ae3200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.128 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.129+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:00.129 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:00.129+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d945f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:01.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:01.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:01.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:01.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.593+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:01.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:01.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:01.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:01.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:01.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:01.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:01.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:01.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:01.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:01.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:01.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:01.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:01.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:01.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:03.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:42:03.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:03.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:03.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:03.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:03.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:03.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:03.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:03.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:03.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:04.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:04.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:04.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:04.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:04.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:04.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:04.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:04.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:04.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:04.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:04.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:04.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:04.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:04.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:06.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:06.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:06.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:06.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:06.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:06.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:06.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:06.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:06.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:06.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:07.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:07.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:07.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:07.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:07.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:07.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:07.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:07.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:07.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:07.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:07.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:07.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:08.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:08.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:08.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:08.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:08.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:08.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:08.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:42:08.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:08.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:08.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:08.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:08.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:08.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:08.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:08.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:08.668 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:08.669 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:08.669 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:42:08.680 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:09.303 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:09.301+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:09.314 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:09.313+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:09.333 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:09.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:09.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:09.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:09.334 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:09.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:09.805 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/263115127 2023-10-14T00:42:09.805 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:42:09.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244129 2023-10-14T00:42:09.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:09.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/263115127 ]] 2023-10-14T00:42:09.806 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/263115127 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\6\3\1\1\5\1\2\7 ]] 2023-10-14T00:42:09.807 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:42:09.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:09.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:09.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:09.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:09.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:09.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:11.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:11.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:11.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:11.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:11.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:11.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:13.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:13.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:13.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:13.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:13.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:13.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:13.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:13.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:13.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:13.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:13.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:13.965+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:13.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:13.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:13.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:13.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:14.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:14.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:14.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:14.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:14.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:14.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:14.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:14.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:15.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:15.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:15.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:15.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:15.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:15.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:15.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:15.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:15.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:15.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:15.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:15.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:15.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:15.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:17.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:17.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:17.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:17.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:17.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:17.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:17.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:17.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:18.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:17.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:18.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:18.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:18.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:18.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:18.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:18.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:18.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:18.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:18.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:18.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:18.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:42:18.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:18.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:18.273 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:18.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:18.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:18.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:18.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:18.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:18.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:18.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:18.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:18.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:19.812 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:42:19.812 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:19.824 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:19.827 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:20.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:20.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:20.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:20.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:20.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:20.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:20.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:20.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:20.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:20.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:20.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:20.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:20.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:20.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:20.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:20.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:20.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:20.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:21.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:21.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:21.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:21.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:21.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:21.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:21.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:21.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:21.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:21.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:21.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:21.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:21.420 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/4079301895 2023-10-14T00:42:21.421 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:42:21.422 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244141 2023-10-14T00:42:21.422 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:21.423 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/4079301895 ]] 2023-10-14T00:42:21.423 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/4079301895 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\2\6\3\1\1\5\1\2\7 ]] 2023-10-14T00:42:21.423 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/4079301895 2023-10-14T00:42:22.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:22.556+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:22.566 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:22.564+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:22.591 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:22.588+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:22.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:22.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:22.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:22.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:22.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:22.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:22.820 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:22.820+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8462240 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:22.859 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:22.856+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d84623c0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:22.897 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:22.896+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8463c80 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:22.939 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:22.936+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8462e40 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:22.965 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:22.964+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8463500 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:23.020 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:23.020+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d84632c0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:23.067 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:23.064+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8463440 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:23.100 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:23.100+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8210900 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:23.105 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:23.104+0000 7f0826b0e640 -1 librbd::mirror::GetInfoRequest: 0x5627d8210fc0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:23.106 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:23.104+0000 7f082630d640 -1 librbd::mirror::GetInfoRequest: 0x5627d8210000 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:23.107 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:42:23.104+0000 7f0823307640 -1 librbd::api::Mirror: image_info_list: failed to list mirror image statuses: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:42:23.800 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/4079301895 until 2023-10-14T01:42:23.159425+0000 (3600 sec) 2023-10-14T00:42:23.894 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/4079301895 2023-10-14T00:42:23.894 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:42:24.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:24.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:24.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:24.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:24.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:24.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:24.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:24.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:24.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:24.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:24.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:24.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:24.584+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:24.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:24.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:24.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:24.826 INFO:tasks.workunit.client.0.smithi138.stderr:listed 43 entries 2023-10-14T00:42:24.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:24.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:24.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:24.889 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/4079301895 2023-10-14T00:42:24.890 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:42:25.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:25.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:25.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:25.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:25.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:25.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:25.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:25.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:25.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:25.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:25.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:25.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:25.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:25.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:25.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:25.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:25.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:25.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:25.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:25.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:25.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:25.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:25.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:25.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:28.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:28.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:28.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:28.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:28.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:28.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:28.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:28.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:42:28.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:42:28.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:28.333 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:28.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:28.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:28.379 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:28.379 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:28.376+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:28.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:28.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:28.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:28.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:28.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:28.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:28.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:30.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:30.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:30.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:30.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:30.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:30.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:30.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:30.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:30.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:30.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:30.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:30.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:31.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:31.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:31.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:31.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:31.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:31.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:31.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:31.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:31.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:31.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:31.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:31.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:31.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:31.952+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:31.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:31.984+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:31.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:31.984+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:31.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:31.984+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:32.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:32.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:32.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:32.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:32.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:32.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:32.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:32.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:32.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:32.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:32.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:32.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:32.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:32.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:33.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:33.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:33.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:33.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:33.467 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:33.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:33.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:33.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:33.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:33.476+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:33.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:33.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:33.497 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:33.496+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:33.513 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:33.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:33.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:33.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:33.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:33.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:33.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:33.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:33.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:33.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:34.894 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:42:34.895 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:34.920 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:36.071 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:42:36.072 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:42:36.073 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244156 2023-10-14T00:42:36.074 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:36.074 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:42:36.074 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\4\0\7\9\3\0\1\8\9\5 ]] 2023-10-14T00:42:36.074 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist add 172.21.15.138:0/393253116 2023-10-14T00:42:36.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:36.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:36.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:36.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:36.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:36.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:36.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:36.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:36.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:36.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:36.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:37.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:37.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:37.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:37.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:37.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:37.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:37.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:37.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:37.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:37.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:37.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:37.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:37.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:37.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:38.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:38.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:38.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:38.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:38.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:38.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:38.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:38.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:38.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:38.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:38.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:38.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:38.821 INFO:tasks.workunit.client.0.smithi138.stderr:blocklisting 172.21.15.138:0/393253116 until 2023-10-14T01:42:38.222066+0000 (3600 sec) 2023-10-14T00:42:38.902 INFO:tasks.workunit.client.0.smithi138.stderr:+ ceph osd blocklist ls 2023-10-14T00:42:38.905 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep -q 172.21.15.138:0/393253116 2023-10-14T00:42:39.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:39.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:39.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:39.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:39.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:39.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:39.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:39.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:39.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:39.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:40.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:40.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:40.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:40.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:40.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:40.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:40.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:40.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:40.056 INFO:tasks.workunit.client.0.smithi138.stderr:listed 44 entries 2023-10-14T00:42:40.115 INFO:tasks.workunit.client.0.smithi138.stderr:+ PREV_CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:42:40.116 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:42:41.418 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:41.416+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:41.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:41.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:42.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:42.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:42.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:42.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:42.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:42.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:42.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:42.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:42.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:42.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:42.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:42.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:43.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:43.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:43.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:42:43.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:43.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:43.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:43.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:43.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:43.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:43.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:43.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:43.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:43.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:43.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:43.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:43.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:43.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:43.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:44.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:44.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:44.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:44.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:44.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:44.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:44.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:44.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:44.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:44.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:44.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:44.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:45.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:42:45.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:45.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:45.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:45.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:45.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:45.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:45.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:45.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:45.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:46.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:46.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:42:46.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:46.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:46.460 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:46.460+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:46.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:46.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:46.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:46.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:46.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:46.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:46.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:46.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:47.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:47.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:47.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:47.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:47.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:47.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:47.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:47.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:47.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:47.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:47.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:47.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:47.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:47.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:47.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:47.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:48.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:48.512+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:48.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:48.536+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:48.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:48.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:48.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:48.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:48.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:48.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:48.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:48.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:50.120 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:42:50.120 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:50.123 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:50.137 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:50.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:50.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:42:50.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:50.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:50.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:50.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:50.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:50.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:50.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:50.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:50.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:50.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:50.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:50.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:50.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:50.235 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:50.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:50.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:50.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:50.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:50.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:50.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:50.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:50.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:50.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:50.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:50.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:50.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:50.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:50.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:50.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:51.471 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:42:51.472 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:42:51.472 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244171 2023-10-14T00:42:51.473 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:51.473 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:42:51.473 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:42:51.473 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:42:51.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:51.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:51.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:51.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:51.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:51.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:51.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:51.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:51.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:51.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:51.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:51.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:52.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:52.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:52.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:52.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:52.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:52.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:52.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:52.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:52.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:52.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:52.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:52.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:52.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:52.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:55.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:55.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:55.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:55.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:55.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:55.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:55.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:55.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:55.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:55.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:55.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:55.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:55.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:55.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:55.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:55.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:56.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:56.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:56.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:56.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:56.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:56.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:56.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:56.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:42:56.820 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:56.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:56.836 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:56.836+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:57.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:57.045 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.044+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:57.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:57.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:57.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:57.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:57.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:42:57.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:42:57.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:42:57.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.444+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:42:57.463 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.460+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:42:57.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:42:57.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:42:57.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:42:57.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:42:59.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:42:59.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:42:59.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:42:59.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:42:59.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:42:59.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:42:59.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:42:59.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:42:59.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.000+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.004 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.000+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9108080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.008+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.008+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.008+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8282800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.011 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.008+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d905d080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.016+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.019 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.016+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d7bfa400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.020 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.016+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.020 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.016+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9358200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.026 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.024+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.026 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.024+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8e17200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.030 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.028+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.031 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.028+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.037 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.036+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.037 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.036+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd8480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.040 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.036+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.040 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.036+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8dd8880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.044+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.044 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.044+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.060 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.056+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.060 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.060+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9461400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.060+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.060+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.060+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.061 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.060+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9460e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.060+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.062 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.060+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d8deb480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.067 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.064+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.068 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.064+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d905e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.071 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.068+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.072 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.068+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9486700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.072+0000 7f0825b0c640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:00.075 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:43:00.072+0000 7f0825b0c640 -1 librbd::ImageState: 0x5627d9487500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:43:01.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:01.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:01.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:01.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:01.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:01.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:01.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:01.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:01.515 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:01.516 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:01.516 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:43:01.539 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:03.165 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:43:03.165 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:43:03.166 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244183 2023-10-14T00:43:03.166 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:03.166 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:43:03.167 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:43:03.167 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:43:03.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:03.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:43:03.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:03.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:03.531 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:03.528+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:04.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:04.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:04.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:04.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:04.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:04.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:04.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:04.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:04.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:04.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:04.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:04.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:04.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:04.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:04.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:04.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:04.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:04.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:04.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:04.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:04.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:04.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:04.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:05.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:05.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:05.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:05.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:05.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:05.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:05.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:05.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:05.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:05.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:06.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:06.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:06.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:06.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:06.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:06.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:06.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:06.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:06.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:06.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:06.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:06.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:07.312 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:07.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:07.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:07.343+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:09.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:09.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:09.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:09.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:09.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:09.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:09.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:09.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:09.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:09.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:09.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:09.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:10.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:10.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:10.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:10.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:10.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:10.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:10.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:10.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:10.608 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:10.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:10.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:10.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:10.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:10.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:10.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:10.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:10.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:10.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:10.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:10.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:10.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:10.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:10.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:10.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:12.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:12.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:12.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:12.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:12.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:12.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:12.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:12.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:12.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:12.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:12.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:12.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:13.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:13.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:13.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:13.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:13.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:13.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:13.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:13.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:13.168 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:43:13.168 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:13.169 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:13.173 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:13.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:13.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:13.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:13.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:13.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:13.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:13.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:13.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:13.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:13.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:14.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:14.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:43:14.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:14.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:14.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:14.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:14.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:14.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:14.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:14.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:14.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:14.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:14.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:14.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:14.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:14.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:14.518 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:43:14.519 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:43:14.520 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244194 2023-10-14T00:43:14.521 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:14.521 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:43:14.521 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:43:14.521 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:43:14.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:14.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:14.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:14.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:14.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:15.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:15.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:43:15.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:15.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:15.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:15.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:18.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:18.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:18.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:18.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:18.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:18.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:18.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:18.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:18.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:18.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:18.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:18.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:19.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:19.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:19.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:19.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:19.644 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:19.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:19.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:19.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:19.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:19.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:19.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:19.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:20.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:20.779+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:20.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:20.795+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:20.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:20.827+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:22.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:22.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:22.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:22.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:22.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:22.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:22.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:22.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:22.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:22.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:22.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:22.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:23.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:23.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:23.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:23.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:23.463 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:23.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:23.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:23.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:23.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:23.491+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:23.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:23.491+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:23.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:23.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:23.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:23.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:23.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:23.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:23.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:24.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:24.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:24.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:24.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:24.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:24.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:24.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:24.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:24.546 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:43:24.551 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:24.552 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:24.556 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:24.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:24.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:24.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:24.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:24.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:24.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:24.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:24.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:24.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:24.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:24.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:24.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:24.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:24.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:24.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:24.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:25.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:25.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:25.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:25.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:25.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:25.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:25.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:25.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:25.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:25.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:25.951 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:43:25.952 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:43:25.952 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244205 2023-10-14T00:43:25.953 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:25.953 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:43:25.953 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:43:25.953 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:43:27.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:27.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:27.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:27.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:27.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:27.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:27.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:27.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:27.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:27.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:27.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:27.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:27.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:27.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:28.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:28.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:28.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:28.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:28.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:28.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:28.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:28.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:28.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:28.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:28.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:28.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:28.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:28.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:29.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:29.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:43:29.719 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:29.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:29.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:29.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:29.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:29.727+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:29.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:29.727+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:29.736 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:29.727+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:31.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:31.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:43:32.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:32.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:43:32.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:32.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:32.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:32.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:32.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:32.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:32.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:32.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:32.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:32.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:32.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:32.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:32.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:32.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:32.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:32.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:32.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:32.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:43:32.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:43:32.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:32.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:32.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:32.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:32.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:32.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:32.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:32.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:32.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:32.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:33.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:33.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:33.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:33.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:33.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:33.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:33.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:33.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:33.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:33.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:33.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:33.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:33.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:33.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:35.957 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:43:35.958 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:35.958 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:35.958 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:36.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:36.863+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:37.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:37.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:37.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:37.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:37.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:37.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:37.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:37.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:37.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:37.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:37.298 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:43:37.299 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:43:37.299 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244217 2023-10-14T00:43:37.300 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:37.300 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:43:37.300 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:43:37.300 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:43:40.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:40.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:40.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:40.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:40.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:40.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:40.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:40.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:43.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:43.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:43.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:43.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:43.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:43.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:43.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:43.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:43.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:43.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:43.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:43.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:43.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:43.727+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:43.741 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:43.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:43.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:43.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:43.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:43.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:43.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:43.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:43.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:43.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:44.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:44.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:44.548 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:44.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:44.548 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:44.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:44.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:44.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:47.306 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:47.307 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:47.311 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:47.317 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:43:48.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:48.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:48.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:48.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:48.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:48.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:48.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:43:48.856 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:43:48.858 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244228 2023-10-14T00:43:48.858 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:48.858 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:43:48.859 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:43:48.859 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:43:49.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:49.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:49.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:49.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:49.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:49.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:49.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:49.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:49.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:49.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:49.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:49.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:49.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:49.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:49.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:49.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:50.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:50.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:50.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:50.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:50.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:50.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:50.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:43:50.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:43:50.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:50.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:50.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:50.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:50.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:50.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:50.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:50.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:53.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.083+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:53.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:53.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:53.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:53.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:43:53.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:53.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:53.718 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:53.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:53.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:53.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:53.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:53.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:43:53.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:53.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:53.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:53.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:53.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:53.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:53.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:53.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:53.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:53.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:54.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:54.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:43:54.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:54.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:54.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:54.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:54.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:54.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:54.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:54.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:54.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:54.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:55.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:55.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:43:55.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:55.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:55.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:55.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:55.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:55.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:55.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:55.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:56.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:55.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:56.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:56.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:43:56.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:56.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:56.014 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:56.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:43:56.014 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:56.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:43:56.014 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:56.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:43:56.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:56.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:56.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:56.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:56.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:56.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:58.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:58.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:43:58.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:58.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:43:58.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:43:58.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:43:58.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:58.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:43:58.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:58.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:43:58.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:58.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:43:58.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:43:58.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:43:58.872 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:58.873 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:58.885 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:58.900 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:44:00.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:00.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:00.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:00.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:00.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:00.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:00.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:00.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:00.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:00.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:00.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:00.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:00.215 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:44:00.216 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:44:00.217 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244240 2023-10-14T00:44:00.217 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:00.217 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:44:00.218 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:44:00.218 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:44:01.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:01.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:44:01.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:01.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:01.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:01.978+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:02.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:02.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:02.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:02.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:02.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:02.015 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:02.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:02.015 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:02.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:02.015 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:02.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:02.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:02.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:02.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:02.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:44:02.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:44:02.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:02.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:02.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:02.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:02.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:02.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:02.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:02.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:03.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:03.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:03.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:03.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:05.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:05.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:05.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:05.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:05.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:05.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:05.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:05.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:05.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:05.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:05.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:05.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:05.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:07.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:07.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:07.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:07.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:07.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:07.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:07.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:07.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:07.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:07.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:07.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:07.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:07.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:07.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:08.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:08.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:44:08.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:08.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:08.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:08.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:08.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:08.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:08.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:08.554+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:08.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:08.554+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:08.560 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:08.554+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:09.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:09.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:09.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:09.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:09.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:09.358+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:09.367 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:09.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:09.388 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:09.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:09.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:09.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:09.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:09.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:09.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:09.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:10.219 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:44:10.219 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:10.219 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:10.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:10.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:10.227 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:10.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:10.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:10.255 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:10.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:10.256 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:10.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:10.410 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:10.406+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:10.437 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:10.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:10.437 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:10.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:11.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:11.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:11.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:11.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:11.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:11.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:11.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:11.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:11.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:11.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:11.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:11.450+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:11.748 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:44:11.750 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:44:11.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244251 2023-10-14T00:44:11.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:11.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:44:11.753 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:44:11.753 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:44:14.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:14.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:14.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:14.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:14.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:14.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:14.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:14.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:14.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:14.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:16.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:16.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:16.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:16.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:16.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:16.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:16.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:16.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:16.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:16.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:16.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:16.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:17.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:17.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:17.802 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:17.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:17.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:17.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:17.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:17.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:17.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:17.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:17.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:17.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:17.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:17.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:18.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:18.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:18.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:18.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:18.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:18.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:18.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:18.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:18.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:18.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:19.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:19.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:19.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:19.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:19.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:19.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:19.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:19.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:19.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:19.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:21.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:21.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:21.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:21.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:21.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:21.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:21.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:21.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:21.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:21.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:21.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:21.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:21.755 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:44:21.755 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:21.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:21.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:21.764 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:21.783 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:22.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:22.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:22.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:22.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:22.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:22.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:22.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:22.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:22.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:22.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:22.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:22.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:22.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:22.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:22.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:23.452 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:44:23.456 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:44:23.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244263 2023-10-14T00:44:23.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:23.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:44:23.457 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:44:23.458 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:44:25.076 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:25.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:25.085 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:25.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:25.139 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:25.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:25.149 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:25.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:25.149 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:25.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:25.149 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:25.146+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:27.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:27.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:27.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:27.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:27.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:27.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:27.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:27.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:28.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:28.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:28.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:28.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:28.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:28.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:29.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:29.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:29.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:29.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:29.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:29.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:29.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:29.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:29.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:29.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:29.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:29.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:30.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:30.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:30.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:30.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:30.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:30.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:30.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:30.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:30.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:30.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:30.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:30.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:30.217 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:30.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:30.230 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:30.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:30.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:30.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:30.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:30.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:30.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:30.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:31.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:31.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:31.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:31.006+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:31.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:31.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:31.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:31.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:31.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:31.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:31.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:31.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:31.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:31.034+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:31.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:31.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:31.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:31.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:31.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:31.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:31.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:31.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:31.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:31.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:31.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:31.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:31.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:31.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:31.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:31.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:31.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:31.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:31.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:31.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:31.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:31.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:32.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:32.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:32.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:32.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:32.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:32.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:32.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:32.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:32.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:33.468 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:33.472 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:33.478 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:44:33.497 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:34.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:34.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:34.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:34.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:34.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:34.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:34.869 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:44:34.870 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:44:34.871 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244274 2023-10-14T00:44:34.872 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:34.872 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:44:34.872 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:44:34.872 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:44:35.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:35.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:36.674 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:36.670+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:36.675 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:36.670+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:36.675 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:36.670+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:37.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:37.378+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:37.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:37.378+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:37.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:37.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:37.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:37.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:37.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:37.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:37.567 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:37.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:38.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:38.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:38.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:38.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:41.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:41.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:41.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:41.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:41.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:41.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:41.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:41.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:41.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:41.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:41.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:41.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:41.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:41.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:41.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:41.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:41.494+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:44.419 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:44.414+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:44.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:44.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:44.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:44.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:44.531 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:44.526+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:44.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:44.526+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:44.532 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:44.526+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:44.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:44.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:44.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:44.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:44.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:44.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:44.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:44.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:44.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:44.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:44.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:44.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:44.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:44.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:44.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:44.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:44.886 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:44:44.887 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:44.887 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:44.906 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:44.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:45.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:45.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:44.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:45.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:45.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:45.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:45.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:45.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:45.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:45.298 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.294+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:45.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:45.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:45.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:45.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:45.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:45.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:45.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:45.798 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:45.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:44:45.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:45.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.806+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:45.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:45.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:45.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:45.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:45.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:45.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:45.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:45.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:45.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:45.967 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:44:45.976 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:44:45.977 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244285 2023-10-14T00:44:45.977 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:45.977 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:44:45.978 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:44:45.978 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:44:47.535 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:47.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:47.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:47.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:47.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:47.590+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:47.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:47.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:47.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:47.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:47.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:47.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:47.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:47.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:47.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:47.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:47.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:47.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:47.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:47.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:47.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:48.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:48.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:48.366 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:48.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:48.366 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:48.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:48.366 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:48.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:50.073 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:50.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:50.110 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:50.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:50.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:50.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:50.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:50.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:50.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:50.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:50.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:50.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:50.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:50.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:52.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:52.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:52.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:52.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:52.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:52.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:52.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:52.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:52.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:52.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:52.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:53.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:53.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:53.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:53.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:44:53.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:53.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:53.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:53.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:53.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:54.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.349+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:44:54.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.349+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:54.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.361+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:44:54.404 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:54.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:44:54.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:44:54.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:44:54.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.705+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:44:54.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:44:54.745+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:44:55.980 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:55.981 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:44:56.006 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:56.012 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:56.802 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:44:56.797+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:44:56.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:56.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:44:56.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:56.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:44:56.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:56.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:56.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:56.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:56.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:56.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:56.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:56.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:57.171 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:44:57.174 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:44:57.176 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244297 2023-10-14T00:44:57.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:57.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:44:57.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:44:57.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:44:57.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:57.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:57.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:57.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:57.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:57.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:57.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:57.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:57.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:57.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:58.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:58.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:58.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:58.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:58.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:58.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:58.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:58.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:58.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:58.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:58.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:58.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:59.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:59.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:59.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:59.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:59.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:44:59.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:44:59.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:44:59.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:44:59.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:44:59.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:44:59.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:44:59.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:01.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:01.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:01.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:01.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:01.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:01.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:01.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:01.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:01.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:01.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:01.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:03.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:03.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:03.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:03.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:03.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:03.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:03.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:03.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:03.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:03.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:04.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:04.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:04.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:04.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:04.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:04.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:04.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:04.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:04.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:04.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:05.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:05.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:05.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:05.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:05.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:05.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:05.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:05.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:05.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:05.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:05.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:05.397+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:45:05.404 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:05.397+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:05.449 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:05.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:05.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:05.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:05.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:05.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:07.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:07.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:07.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:07.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:07.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:07.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:07.186 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:45:07.187 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:07.194 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:07.196 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:08.692 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:45:08.693 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:45:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244308 2023-10-14T00:45:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:45:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:45:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:45:09.755 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:09.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:09.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:09.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:09.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:09.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:09.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:09.789+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:09.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:09.789+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:09.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:09.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:09.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:09.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:09.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:09.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:09.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:09.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:09.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:09.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:11.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:11.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:11.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:11.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:11.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:11.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:11.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:11.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:11.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:11.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:11.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:12.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:12.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:12.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:12.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:12.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:12.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:12.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:12.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:12.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:12.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:12.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:12.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:12.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:12.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:13.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:13.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:13.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:13.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:13.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:13.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:13.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:13.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:14.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:14.469+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:14.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:14.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:14.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:14.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:14.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:14.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:14.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:14.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:14.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:14.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:14.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:14.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:14.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:14.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:14.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:14.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:14.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:14.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:14.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:14.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:14.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:14.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:15.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:15.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:15.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:15.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:15.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:15.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:15.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:15.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:15.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:15.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:16.764 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:16.761+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:16.764 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:16.761+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:45:16.764 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:16.761+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:16.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:16.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:16.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:16.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:16.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:16.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:16.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:16.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:17.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:17.885+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:17.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:17.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:17.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:17.937+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:45:17.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:17.937+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:18.699 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:45:18.699 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:18.700 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:18.700 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:18.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:18.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:18.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:18.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:18.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:18.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:18.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:18.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:18.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:18.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:18.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:18.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:19.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:19.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:19.572 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:19.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:20.085 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:45:20.085 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:45:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244320 2023-10-14T00:45:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:45:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:45:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:45:22.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:22.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:22.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:22.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:22.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:22.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:22.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:22.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:22.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:22.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:23.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:23.705+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:23.706 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:23.705+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:45:23.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:23.709+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:23.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:23.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:23.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:23.729+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:45:23.736 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:23.737+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:24.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:24.021+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:24.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:24.029+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:45:24.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:24.041+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:24.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:24.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:24.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:24.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:45:24.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:24.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:24.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:24.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:24.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:24.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:24.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:24.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:24.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:24.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:24.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:24.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:25.103 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:25.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:45:25.110 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:25.109+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:25.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:25.137+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:25.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:25.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:45:25.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:25.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:28.896 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:45:28.897+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:45:29.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:29.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:45:29.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:29.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:29.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:29.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:30.088 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:30.089 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:45:30.090 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:31.449 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:45:31.466 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:45:31.467 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244331 2023-10-14T00:45:31.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:31.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:45:31.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:45:31.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:45:32.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:32.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:45:32.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:32.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:32.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:32.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:45:32.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:32.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:32.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:32.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:32.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:32.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:45:32.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:32.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:33.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:33.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:33.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:33.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:33.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:33.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:33.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:45:33.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:33.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:33.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:33.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:33.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:33.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:33.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:33.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:34.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:34.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:34.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:34.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:34.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:34.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:34.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:34.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:34.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:34.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:34.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:34.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:35.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:35.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:35.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:35.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:35.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:35.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:35.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:35.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:35.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:35.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:35.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:35.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:35.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:35.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:36.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:36.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:36.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:36.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:37.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:37.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:37.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:37.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:37.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:37.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:37.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:37.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:37.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:37.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:37.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:37.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:37.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:38.449 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:38.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:45:41.491 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:45:41.491 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:41.507 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:41.520 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:41.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:41.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:41.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:41.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:41.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:41.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:41.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:41.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:41.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:41.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:42.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:42.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:42.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:42.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:42.911 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:45:42.920 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:45:42.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244342 2023-10-14T00:45:42.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:42.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:45:42.920 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:45:42.921 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:45:42.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:42.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:42.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:42.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:42.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:42.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:42.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:42.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:42.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:42.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:43.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:43.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:43.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:43.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:43.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:43.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:43.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:43.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:43.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:43.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:43.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:43.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:43.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:43.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:43.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:43.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:43.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:43.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:43.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:43.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:43.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:43.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:43.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:43.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:44.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:44.868+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:44.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:44.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:47.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:47.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:45:47.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:47.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:45:47.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:47.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:45:47.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:47.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:47.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:47.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:47.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:47.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:47.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:47.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:48.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:48.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:48.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:48.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:48.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:48.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:48.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:48.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:48.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:48.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:49.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:49.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:49.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:49.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:49.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:49.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:49.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:49.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:49.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:49.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:50.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:45:50.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:50.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:50.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:50.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:50.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:50.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:50.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:50.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:52.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:52.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:45:52.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:52.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:52.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:52.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:52.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:52.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:52.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:52.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:52.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:52.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:52.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:52.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:52.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:52.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:52.933 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:52.933 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:45:52.955 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:52.956 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:54.352 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:45:54.354 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:45:54.355 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244354 2023-10-14T00:45:54.355 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:54.356 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:45:54.356 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:45:54.356 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:45:57.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:57.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:57.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:57.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:57.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:57.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:57.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:57.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:57.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:58.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:58.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:58.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:58.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:58.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:58.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:58.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:58.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:58.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:58.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:45:58.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:58.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:45:58.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:58.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:45:58.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:45:58.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:45:59.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:59.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:59.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:59.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:59.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:59.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:45:59.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:45:59.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:45:59.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:45:59.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:45:59.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:45:59.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:45:59.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:45:59.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:45:59.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:00.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:00.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:46:00.358 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:00.356+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:00.366 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:00.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:00.374 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:00.372+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:00.393 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:00.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:00.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:00.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:00.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:01.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:01.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:01.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:01.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:01.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:01.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:01.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:01.772 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:46:01.768+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:46:01.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:01.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:01.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:01.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:01.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:02.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:02.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:02.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:02.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:02.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:02.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:46:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:02.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:02.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:02.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:02.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:02.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:02.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:03.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:03.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:03.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:03.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:03.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:03.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:03.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:03.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:04.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:04.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:04.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:04.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:04.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:04.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:04.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:04.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:04.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:04.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:04.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:04.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:04.370 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:04.381 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:04.394 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:04.397 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:46:05.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:46:05.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:05.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:05.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:05.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:05.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:05.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:05.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:05.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:05.822 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:46:05.822 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:46:05.823 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244365 2023-10-14T00:46:05.823 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:05.823 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:46:05.823 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:46:05.824 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:46:06.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:06.448+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:06.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:06.456+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:06.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:06.472+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:06.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:06.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:06.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:06.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:06.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:06.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:06.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:06.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:06.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:06.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:08.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:46:08.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:46:08.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:08.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:08.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:08.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:08.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:08.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:08.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:08.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:08.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:09.908 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:09.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:09.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:09.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:09.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:09.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:09.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:09.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:09.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:09.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:10.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:10.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:10.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:10.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:10.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:10.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:10.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:10.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:10.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:10.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:10.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:10.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:10.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:10.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:10.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:10.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:10.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:10.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:10.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:10.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:11.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:11.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:11.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:11.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:11.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:11.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:11.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:11.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:11.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:12.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:12.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:12.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:12.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:12.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:12.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:12.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:12.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:13.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:13.744+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:13.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:13.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:13.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:13.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:13.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:13.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:13.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:13.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:13.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:13.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:14.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:14.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:14.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:14.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:14.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:14.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:14.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:14.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:14.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:14.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:15.833 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:15.837 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:15.845 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:46:15.869 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:17.226 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:46:17.227 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:46:17.229 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244377 2023-10-14T00:46:17.229 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:17.229 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:46:17.230 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:46:17.230 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:46:18.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.120+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:18.130 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.128+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:18.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.128+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:18.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.128+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:18.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.128+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:18.250 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:18.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:18.361 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.360+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:18.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.372+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:18.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:18.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:18.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:18.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:19.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:19.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:20.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:20.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:20.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:20.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:20.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:20.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:20.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:20.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:20.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:20.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:20.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:20.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:21.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:21.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:21.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:21.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:21.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:21.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:22.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:22.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:22.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:22.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:22.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:22.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:22.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:22.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:23.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:23.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:23.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:23.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:23.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:23.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:23.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:23.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:23.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:23.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:23.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:23.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:25.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:25.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:25.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:25.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:25.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:25.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:25.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:25.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:25.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:25.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:25.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:25.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:26.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:26.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:26.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:26.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:26.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:26.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:26.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:26.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:26.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:26.510 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:26.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:46:26.510 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:26.508+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:26.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:26.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:26.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:26.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:26.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:26.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:26.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:26.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:26.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:26.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:26.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:26.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:26.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:26.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:26.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:26.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:26.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:26.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:26.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:26.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:27.233 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:46:27.233 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:27.235 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:27.270 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:27.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:27.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:27.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:27.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:27.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:27.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:28.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:28.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:28.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:28.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:28.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:28.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:28.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:28.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:28.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:28.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:28.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:28.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:28.586 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:46:28.594 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:46:28.595 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244388 2023-10-14T00:46:28.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:28.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:46:28.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:46:28.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:46:28.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:28.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:28.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:28.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:28.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:28.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:28.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:28.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:28.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:28.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:28.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:28.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:29.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:29.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:29.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:29.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:29.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:29.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:29.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:29.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:29.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:32.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:32.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:32.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:32.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:32.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:32.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:32.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:32.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:32.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:32.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:32.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:32.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:33.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:33.280+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:46:33.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:33.280+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:33.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:33.304+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:33.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:33.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:33.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:33.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:33.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:33.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:33.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:33.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:34.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:34.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:34.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:34.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:34.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:34.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:34.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:34.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:34.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:34.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:34.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:34.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:34.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:34.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:35.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:35.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:46:35.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:35.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:35.876 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:35.872+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:35.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:35.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:35.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:35.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:35.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:35.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:35.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:35.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:36.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:36.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:36.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:36.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:36.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:36.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:36.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:36.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:36.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:36.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:36.311 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.308+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:36.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:36.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:36.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:36.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:36.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:36.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:36.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:36.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.604+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:46:36.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.604+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:36.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:36.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:36.640+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:37.026 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:37.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:37.048 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:37.044+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:37.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:37.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:46:37.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:37.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:37.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:37.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:38.606 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:38.607 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:38.608 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:46:38.624 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:39.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:39.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:39.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:39.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:40.175 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:46:40.176 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:46:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244400 2023-10-14T00:46:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:46:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:46:40.185 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:46:40.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:40.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:46:40.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:40.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:40.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:40.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:40.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:40.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:40.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:40.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:40.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:40.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:41.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:41.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:41.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:41.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:41.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:41.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:42.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:42.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:42.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:42.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:42.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:42.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:42.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:42.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:42.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:42.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:43.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:43.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:43.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:43.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:43.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:43.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:43.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:43.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:43.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:43.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:43.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:43.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:43.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:43.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:44.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:44.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:44.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:44.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:44.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:44.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:44.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:44.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:44.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:44.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:44.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:44.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:45.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:45.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:45.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:45.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:45.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:45.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:45.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:45.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:45.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:45.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:45.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:45.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:47.734 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:47.731+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:46:47.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:47.731+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:47.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:47.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:47.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:47.759+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:47.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:47.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:47.766 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:47.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:47.766 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:47.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:48.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:48.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:48.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:48.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:48.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:48.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:48.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:48.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:48.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:48.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:48.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:48.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:48.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:50.193 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:50.193 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:50.210 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:46:50.231 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:51.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:51.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:51.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:51.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:51.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:51.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:51.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:51.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:51.633 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:46:51.633 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:46:51.634 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244411 2023-10-14T00:46:51.634 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:51.634 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:46:51.634 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:46:51.635 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:46:51.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:51.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:52.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:52.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:52.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:52.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:52.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:52.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:52.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:52.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:52.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:52.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:52.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:52.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:52.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:52.971 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:52.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:46:52.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:52.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:46:53.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:53.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:53.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:53.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:53.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:53.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:53.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:53.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:54.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:54.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:54.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:54.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:54.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:54.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:54.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:54.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:54.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:54.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:54.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:54.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:54.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:55.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:55.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:46:55.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:55.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:46:55.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:55.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:46:55.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:55.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:55.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:55.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:55.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:55.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:55.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:55.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:46:55.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:55.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:46:55.675 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:55.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:46:55.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:55.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:46:55.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:46:55.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:46:57.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:57.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:57.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:57.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:57.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:57.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:58.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:58.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:46:58.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:58.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:46:58.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:58.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:46:58.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:58.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:46:58.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:46:58.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:01.636 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:47:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:01.660 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:01.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:01.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:01.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:01.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:01.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:01.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:01.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:01.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:01.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:03.032 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:47:03.032 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:47:03.033 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244423 2023-10-14T00:47:03.033 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:03.033 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:47:03.033 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:47:03.033 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:47:05.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:05.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:05.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:05.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:05.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:05.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:05.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:05.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:05.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:05.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:05.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:05.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:05.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:05.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:05.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:05.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:05.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:05.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:05.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:05.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:05.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:05.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:05.986 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:05.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:06.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:06.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:06.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:06.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:06.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:06.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:06.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:06.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:06.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:07.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:07.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:07.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:07.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:07.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:07.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:07.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:07.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:07.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:07.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:07.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:07.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:07.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:07.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:12.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:12.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:12.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:12.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:12.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:12.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:12.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:12.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:12.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:12.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:13.039 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:47:13.045 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:13.045 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:13.051 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:13.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:13.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:13.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:13.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:13.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:13.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:13.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:13.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:13.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:14.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:14.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:14.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:14.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:14.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:14.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:14.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:47:14.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:14.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:14.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:14.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:14.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:14.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:14.427 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:47:14.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:14.432 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:47:14.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:14.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:14.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:14.458 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244434 2023-10-14T00:47:14.459 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:14.459 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:47:14.459 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:47:14.459 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:47:16.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:16.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:16.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:16.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:16.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:16.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:16.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:16.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:16.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:16.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:16.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:16.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:16.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:16.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:16.708 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:16.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:16.709 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:16.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:17.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:17.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:17.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:17.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:17.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:17.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:17.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:17.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:17.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:17.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:17.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:17.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:19.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:19.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:19.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:19.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:19.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:19.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:19.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:19.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:19.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:19.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:19.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:19.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:19.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:19.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:19.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:19.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:19.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:19.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:19.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:19.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:19.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:19.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:19.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:19.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:19.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:19.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:19.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:19.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:20.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:20.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:20.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:20.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:20.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:20.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:20.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:20.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:20.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:20.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:22.002 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:21.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:22.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:47:22.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:22.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:22.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:22.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:22.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:22.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:22.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:22.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:22.432 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.431+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:22.432 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.431+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:22.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:22.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.459+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:22.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:22.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:22.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:22.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:22.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:22.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:22.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:22.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:22.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:22.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:22.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:22.907 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:22.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:23.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:23.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:23.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:23.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:23.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:23.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:23.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:23.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:24.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:24.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:24.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:24.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:24.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:24.461 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:47:24.466 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:24.469 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:24.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:24.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:24.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:24.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:24.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:24.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:25.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:47:25.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:47:25.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:25.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:25.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:25.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:25.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:25.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:25.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:25.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:25.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:25.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:25.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:25.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:25.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:25.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:25.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:25.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:25.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:25.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:25.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:25.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:25.631+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:25.878 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:47:25.879 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:47:25.880 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244445 2023-10-14T00:47:25.880 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:25.880 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:47:25.880 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:47:25.881 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:47:27.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:27.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:27.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:27.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:27.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:27.147+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:27.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:27.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:27.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:27.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:27.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:27.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:27.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:27.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:27.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:27.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:27.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:27.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:27.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:27.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:29.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:29.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:29.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:29.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:29.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:29.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:29.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:29.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:29.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:29.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:29.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:29.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:31.066 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:31.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:31.712 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:31.711+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:31.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:31.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:31.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:31.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:32.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:32.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:32.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:32.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:32.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:32.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:32.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:32.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:33.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:33.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:33.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:33.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:33.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:33.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:33.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:33.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:33.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:33.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:33.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:33.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:33.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:33.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:33.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:33.846+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:33.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:33.854+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:33.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:33.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:33.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:33.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:33.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:33.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:34.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:34.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:34.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:34.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:34.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:34.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:34.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:34.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:34.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:34.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:34.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:34.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:34.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:34.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:35.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:35.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:35.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:35.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:35.139 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:35.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:35.141 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:35.138+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:35.885 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:35.889 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:35.889 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:35.891 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:47:36.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:36.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:36.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:36.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:36.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:36.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:36.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:36.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:36.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:36.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:36.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:36.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:36.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:36.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:37.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:37.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:37.333 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:37.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:37.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:37.330+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:37.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:37.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:37.357 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:47:37.366 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:47:37.367 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244457 2023-10-14T00:47:37.367 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:37.368 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:47:37.368 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:47:37.368 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:47:37.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:37.378+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:37.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:37.378+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:37.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:37.378+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:38.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:38.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:38.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:38.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:38.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:38.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:38.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:38.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:38.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:38.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:38.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:38.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:38.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:38.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:38.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:38.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:39.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:39.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:39.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:39.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:39.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:39.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:39.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:39.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:39.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:39.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:40.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:40.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:40.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:40.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:47:40.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:40.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:40.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:40.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:40.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:40.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:40.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:40.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:40.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:42.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:42.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:42.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:42.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:42.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:42.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:42.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:42.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:42.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:42.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:42.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:42.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:42.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:42.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:44.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:44.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:44.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:44.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:44.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:44.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:44.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:44.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:44.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:45.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:45.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:45.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:45.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:45.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:45.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:45.121 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:45.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.886+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:45.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:45.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:45.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:45.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:45.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:47.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:47.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:47.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:47.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:47.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:47.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:47.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:47.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:47.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:47.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:47.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:47.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:47.385 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:47:47.385 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:47.394 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:47.401 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:47.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:47.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:47.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:47.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:47.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:47.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:47.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:47.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:47.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:47.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:47.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:47.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:47.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:47.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:47.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:47.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:47.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:48.768 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:47:48.784 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:47:48.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244468 2023-10-14T00:47:48.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:48.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:47:48.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:47:48.787 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:47:49.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:49.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:49.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:49.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:49.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:49.626 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:49.622+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:49.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:49.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:49.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:49.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:49.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:49.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:49.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:49.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:49.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:49.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:49.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:49.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:49.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:49.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:49.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:49.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:50.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:50.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:50.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:50.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:50.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:50.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:50.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:50.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:50.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:50.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:50.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:50.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:50.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:50.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:50.686 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:50.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:50.687 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:50.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:50.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:50.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:50.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:50.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:50.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:50.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:47:50.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:50.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:47:50.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:50.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:47:51.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:51.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:51.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:51.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:51.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:51.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:51.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:51.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:51.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:51.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:51.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:51.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:52.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:52.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:52.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:52.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:52.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:52.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:52.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:52.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:52.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:52.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:52.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:52.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:53.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:53.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:53.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:53.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:53.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:53.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:53.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:53.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:53.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:53.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:53.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:53.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:53.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:53.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:53.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:53.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:53.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:55.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:55.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:55.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:55.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:55.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:47:55.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:55.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:55.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:55.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:55.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:55.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:56.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:56.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:56.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:56.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:56.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:56.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:56.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:56.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:56.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:47:56.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:56.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:56.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:56.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:56.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:56.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:56.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:56.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:57.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:57.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:57.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:57.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:57.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:57.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:57.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:57.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:57.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:57.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:57.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:57.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:58.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:58.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:47:58.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:58.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:58.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:58.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:58.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:58.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:58.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:58.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:47:58.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:58.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:47:58.801 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:58.801 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:58.803 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:47:58.819 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:59.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:59.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:47:59.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:59.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:47:59.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:59.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:47:59.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:47:59.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:47:59.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:59.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:47:59.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:59.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:47:59.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:59.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:47:59.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:59.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:47:59.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:47:59.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:00.170 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:48:00.176 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:48:00.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244480 2023-10-14T00:48:00.177 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:00.178 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:48:00.178 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:48:00.178 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:48:00.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:00.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:48:00.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:00.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:00.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:00.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:00.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:00.770+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:00.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:00.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:00.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:00.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:00.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:00.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:01.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:01.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:01.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:01.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:01.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:01.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:01.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:02.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:02.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:02.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:02.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:02.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:02.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:02.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:02.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:02.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:02.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:02.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:02.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:02.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:03.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:03.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:03.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:03.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:03.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:03.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:03.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:03.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:04.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:04.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:04.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:04.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:04.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:04.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:04.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:04.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:04.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:04.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:04.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:04.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:06.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:06.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:06.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:06.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:06.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:06.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:06.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:06.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:06.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:06.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:06.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:06.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:06.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:06.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:06.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:08.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:08.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:08.250 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:08.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:08.267 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:08.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:08.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:08.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:08.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:08.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:08.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:08.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:09.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:09.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:09.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:09.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:09.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:09.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:09.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:09.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:09.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:09.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:09.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:09.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:09.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:09.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:10.191 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:10.196 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:48:10.207 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:10.207 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:11.482 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:11.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:11.819 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:48:11.819 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:48:11.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244491 2023-10-14T00:48:11.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:11.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:48:11.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:48:11.821 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:48:13.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:13.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:48:13.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:13.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:13.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:13.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:13.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:13.322+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:13.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:13.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:13.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:13.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:13.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:13.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:13.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:13.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:13.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:13.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:13.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:13.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:13.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:13.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:13.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:13.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:13.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:13.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:13.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:13.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:13.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:13.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:14.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:14.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:14.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:14.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:14.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:14.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:14.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:14.774+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:14.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:14.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:14.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:14.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:14.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:14.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:14.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:14.814+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:14.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:14.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:14.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:14.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:14.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:14.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:14.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:15.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:15.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:15.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:15.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:15.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:15.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:15.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:15.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:15.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:15.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:15.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:15.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:17.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:17.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:18.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:18.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:18.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:18.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:18.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:18.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:18.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:18.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:18.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:18.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:21.825 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:21.838 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:21.841 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:21.860 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:48:23.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:23.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:23.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:23.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:23.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:23.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:23.331 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:48:23.332 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:48:23.339 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244503 2023-10-14T00:48:23.339 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:23.339 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:48:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:48:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:48:23.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:23.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:23.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:23.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:23.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:23.984 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:23.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:48:23.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:23.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:24.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:24.037+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:24.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:24.073+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:24.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:24.085+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:24.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:24.085+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:24.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:24.085+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:25.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:25.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:25.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:25.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:25.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:25.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:25.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:25.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:25.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:25.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:25.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:25.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:26.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:26.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:26.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:26.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:26.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:26.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:26.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:26.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:26.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:26.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:26.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:26.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:27.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:27.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:27.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:27.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:27.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:27.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:27.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:27.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:27.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:27.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:27.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:27.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:27.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:28.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:28.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:28.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:28.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:28.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:28.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:28.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:28.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:28.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:28.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:28.529+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:28.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:28.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:28.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:28.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:29.875 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:29.869+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:29.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:29.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:29.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:29.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:29.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:29.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:29.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:29.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:30.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:30.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:30.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:30.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:30.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:30.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:30.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:30.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:30.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:30.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:30.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:30.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:31.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:31.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:31.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:31.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:31.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:31.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:31.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:31.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:31.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:31.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:31.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:31.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:32.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:32.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:48:32.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:32.113+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:32.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:32.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:32.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:32.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:33.344 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:48:33.344 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:33.345 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:33.348 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:34.778 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:48:34.784 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:48:34.785 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244514 2023-10-14T00:48:34.785 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:34.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:48:34.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:48:34.786 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:48:35.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:35.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:35.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:35.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:35.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:35.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:35.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:35.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:35.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:35.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:35.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:35.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:35.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:35.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:35.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:35.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:35.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:35.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:35.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:35.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:35.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:35.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:35.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:35.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:35.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:35.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:35.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:35.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:35.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:35.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:35.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:35.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:35.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:35.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:36.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:36.621+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:36.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:36.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:36.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:36.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:36.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:36.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:36.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:36.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:37.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:37.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:37.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:37.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:37.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:37.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:37.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:37.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:37.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:37.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:37.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:37.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:38.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:38.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:38.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:38.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:38.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:38.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:38.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:38.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:38.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:38.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:38.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:38.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:38.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:38.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:39.070 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:48:39.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:48:39.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:39.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:39.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:39.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:39.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:39.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:39.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:39.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:39.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:39.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:39.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:39.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:39.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:39.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:39.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:39.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:39.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:39.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:40.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:40.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:48:40.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:40.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:40.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:40.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:40.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:40.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:40.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:40.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:40.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:40.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:41.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.029+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:41.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.029+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:41.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.029+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:41.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.029+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:41.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.029+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:41.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.029+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:41.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:48:41.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:41.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.981+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:41.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:41.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:42.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:42.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:42.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:42.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.245+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:42.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.277+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:42.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:42.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:42.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:42.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:44.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:44.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:44.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:44.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:44.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:44.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:44.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:44.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:44.798 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:48:44.798 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:44.804 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:44.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:45.272 INFO:tasks.workunit.client.0.smithi138.stderr:2023-10-14T00:48:45.269+0000 7f49a7fff640 0 --2- 172.21.15.138:0/193707828 >> v2:172.21.15.138:3300/0 conn(0x7f49b00710e0 0x7f49b0089710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-14T00:48:46.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:46.213+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:46.242 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:46.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:46.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:46.273+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:46.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:46.273+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:46.487 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:48:46.493 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:48:46.508 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244526 2023-10-14T00:48:46.508 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:46.508 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:48:46.508 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:48:46.509 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:48:47.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:47.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:47.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:47.689+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:47.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:47.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:47.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:47.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:47.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:47.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:47.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:47.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:48.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:48:48.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:48:48.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:48.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:48.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:48.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:48.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:48.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:48.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:48.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:51.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:51.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:51.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:51.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:51.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:51.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:51.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:51.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:48:51.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:48:51.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:48:51.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:51.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:51.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:51.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:51.870 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:51.869+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:51.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:51.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:48:51.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:51.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:48:51.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:51.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:48:51.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:51.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:48:51.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:51.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:48:55.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:55.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:55.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:55.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:55.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:55.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:56.528 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:56.532 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:48:56.535 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:56.540 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:57.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:57.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:57.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:57.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:58.079 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:48:58.080 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:48:58.080 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244538 2023-10-14T00:48:58.080 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:58.081 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:48:58.081 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:48:58.081 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:48:59.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:59.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:59.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:59.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:59.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:48:59.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:48:59.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:48:59.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:48:59.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:48:59.445 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:59.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:48:59.445 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:59.445+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:48:59.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:48:59.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:00.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:00.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:01.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:01.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:02.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:49:02.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:02.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:02.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:02.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:02.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:02.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:02.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:02.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:02.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:02.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:02.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:02.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:02.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:02.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:02.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:02.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:02.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:02.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:02.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:02.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:03.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:02.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:03.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:03.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:03.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:03.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:03.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:03.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:03.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:03.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:03.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:03.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:03.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:03.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:03.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:03.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:05.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:05.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:05.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:05.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:05.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:05.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:05.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:05.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:07.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:07.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:07.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:07.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:07.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:07.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:07.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:07.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:08.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:08.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:08.087 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:08.094 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:49:08.094 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:08.098 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:08.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:08.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:08.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:08.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:08.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:08.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:08.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:08.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:08.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:08.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:08.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:08.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:09.461 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:49:09.466 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:49:09.467 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244549 2023-10-14T00:49:09.467 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:09.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:49:09.470 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:49:09.470 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:49:09.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:09.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:09.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:09.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:11.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:49:11.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:11.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:11.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:11.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:11.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:11.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:11.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:11.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:11.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:11.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:11.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:11.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:11.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:11.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:11.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:11.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:11.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:11.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:11.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.744+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:11.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:11.744+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:13.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:13.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:13.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:13.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:49:13.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:13.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:13.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:13.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:13.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:13.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:13.043 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:13.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:13.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:13.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:13.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:13.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:13.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:13.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:13.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:13.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:13.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:13.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:14.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:14.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:14.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:14.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:14.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:14.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:14.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:14.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:14.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:14.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:14.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:14.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:14.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:15.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:15.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:15.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:15.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:49:15.763 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:15.760+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:15.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:15.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:15.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:15.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:15.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:15.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:16.299 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:16.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:16.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:16.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:49:16.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:16.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:16.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:16.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:16.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:16.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:16.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:16.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:16.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:16.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:16.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:16.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:16.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:16.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:16.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:16.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:19.472 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:49:19.472 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:19.486 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:19.504 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:19.572 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:19.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:49:19.573 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:19.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:19.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:19.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:19.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:19.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:19.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:19.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:19.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:19.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:19.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:19.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:19.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:19.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:19.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:19.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:19.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:19.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:19.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:20.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:20.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:20.902 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:49:20.916 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:49:20.917 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244560 2023-10-14T00:49:20.917 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:20.918 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:49:20.918 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:49:20.918 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:49:21.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:21.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:21.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:21.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:21.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:49:21.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:21.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:21.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:21.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:21.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:21.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:21.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:21.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:21.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:21.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:22.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:22.452+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:49:22.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:22.452+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:22.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:22.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:22.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:22.480+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:49:22.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:22.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:22.492+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:22.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:22.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:22.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:22.528+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:22.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:22.528+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:22.531 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:22.528+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:22.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:22.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:22.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:22.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:22.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:22.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:22.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:22.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:22.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:22.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:22.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:22.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:22.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:22.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:22.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:22.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:22.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:22.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:22.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:22.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:22.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:22.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:22.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:22.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:23.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:23.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:23.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:23.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:23.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:23.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:23.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:23.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:24.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:24.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:24.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:24.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:25.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:25.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:25.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:25.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:26.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:26.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:26.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:26.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:26.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:26.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:26.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:26.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:26.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:26.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:26.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:26.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:26.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:26.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:27.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:27.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:27.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:27.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:27.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:27.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:27.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:27.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:28.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:28.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:49:28.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:28.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:49:28.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:28.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:28.907 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:28.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:28.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:28.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:28.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:28.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:28.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:28.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:28.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:28.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:28.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:28.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:29.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:29.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:29.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:29.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:29.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:29.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:29.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:29.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:29.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:29.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:29.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:29.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:49:29.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:29.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:29.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:29.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:30.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:30.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:30.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:30.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:30.919 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:30.939 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:30.941 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:30.942 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:49:32.198 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:49:32.198 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:49:32.206 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244572 2023-10-14T00:49:32.206 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:32.206 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:49:32.207 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:49:32.207 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:49:33.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:33.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:33.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:33.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:33.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:33.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:33.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:33.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:33.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:33.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:33.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:33.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:33.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:33.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:35.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:35.120+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:35.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:35.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:35.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:49:35.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:35.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:35.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:35.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:35.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:35.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:35.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:35.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:35.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:35.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:36.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:36.836+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:36.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:36.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:37.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:37.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:37.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:37.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:37.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:37.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:37.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:37.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:37.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:37.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:37.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:37.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:37.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:37.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:37.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:49:37.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:37.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:37.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:37.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:37.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:37.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:49:37.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:37.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:37.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:37.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:37.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:37.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:38.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:38.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:38.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:38.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:38.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:38.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:38.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:38.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:38.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:38.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:38.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:38.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:38.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:38.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:39.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:39.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:39.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:39.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:39.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:39.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:39.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:39.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:39.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:40.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:40.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:40.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:40.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:40.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:40.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:40.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:40.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:40.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:40.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:40.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:40.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:40.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:40.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:40.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:40.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:40.080 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:40.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:40.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:40.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:40.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:40.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:42.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:42.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:49:42.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:42.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:42.214 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:49:42.215 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:42.215 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:42.216 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:42.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:42.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:42.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:42.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:42.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:42.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:42.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:42.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:43.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:43.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:49:43.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:43.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:43.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:43.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:43.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:43.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:43.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:43.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:43.717 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:49:43.752 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:49:43.753 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244583 2023-10-14T00:49:43.753 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:43.753 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:49:43.753 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:49:43.754 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:49:43.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:43.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:49:43.993 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:43.992+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:44.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:44.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:44.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:44.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:44.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:44.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:44.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:44.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:44.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:44.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:44.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:44.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:44.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:44.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:45.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:45.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:45.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:45.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:45.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:45.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:45.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:45.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:45.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:45.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:45.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:45.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:45.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:45.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:45.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:45.448+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:45.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:45.488+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:45.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:45.536+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:45.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:45.552+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:45.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:45.552+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:49.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:49.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:49.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:49.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:49.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:49.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:49.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:49.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:49.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:49.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:49.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:49.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:49.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:49.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:49.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:49.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:49.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:49.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:49.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:49.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:49.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:51.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:51.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:51.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:51.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:51.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:51.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:52.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:52.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:52.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:52.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:52.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:52.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:52.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:52.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:52.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:52.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:52.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:52.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:52.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:53.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:53.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:53.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:53.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:53.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:53.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:53.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:53.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:53.755 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:53.759 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:49:53.764 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:53.769 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:53.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:53.851+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:53.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:53.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:54.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:54.227+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:49:54.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:54.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:54.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:54.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:49:54.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:54.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:49:54.303 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:54.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:49:54.672 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:54.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:49:54.712 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:49:54.711+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:49:55.048 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:49:55.048 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:49:55.048 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244595 2023-10-14T00:49:55.049 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:55.049 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:49:55.049 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:49:55.049 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:49:57.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:57.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:57.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:57.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:57.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:57.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:57.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:57.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:57.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:57.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:57.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:57.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:58.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:49:58.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:49:58.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:49:58.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:58.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:58.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:49:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:59.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:49:59.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:59.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:49:59.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:59.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:49:59.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:59.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:49:59.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:49:59.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:01.225 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:01.223+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:01.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:01.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:01.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:01.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:01.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:01.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:01.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:01.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:01.298 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:01.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:02.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:02.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:02.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:02.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:02.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:02.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.515+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:02.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:02.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:02.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:04.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:04.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:04.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:04.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:04.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:04.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:04.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:04.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:04.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:04.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:04.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:04.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:04.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:04.479+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:04.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:04.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:04.535 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:04.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:04.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:04.539+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:05.052 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:50:05.059 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:05.059 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:05.076 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:06.581 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:50:06.581 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:50:06.582 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244606 2023-10-14T00:50:06.582 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:06.582 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:50:06.582 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:50:06.583 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:50:07.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:07.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:07.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:07.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:07.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:07.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:07.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:07.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:07.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:07.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:07.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:07.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:07.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:07.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:07.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:08.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:50:08.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:08.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:08.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:08.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:08.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:08.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:08.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:08.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:08.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:08.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:08.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:08.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:08.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:08.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:10.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:50:10.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:10.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:10.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:10.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:10.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:10.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:10.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:10.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:11.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:11.283+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:11.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:11.283+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:11.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:11.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:11.298 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:11.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:11.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:11.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:11.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:11.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:11.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:11.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:12.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:12.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:12.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:12.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:12.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:12.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:12.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:12.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:14.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:14.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:14.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:14.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:14.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:14.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:14.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:14.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:14.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:14.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:15.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:15.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:15.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:15.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:15.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:15.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:15.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:15.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:15.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:15.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:15.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:15.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:15.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:15.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:15.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:15.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:15.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:15.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:15.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:15.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:15.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:15.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:15.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:15.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:15.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:15.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:16.593 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:50:16.605 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:16.612 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:16.613 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:17.909 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:50:17.910 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:50:17.910 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244617 2023-10-14T00:50:17.910 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:17.910 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:50:17.911 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:50:17.911 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:50:18.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:18.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:18.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:18.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:18.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:18.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:18.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:18.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:18.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:18.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:18.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:18.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:18.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:18.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:18.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:18.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:18.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:18.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:18.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:18.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:18.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:18.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:18.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:18.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:18.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:19.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:50:19.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:19.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:19.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:19.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:19.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:19.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:19.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:19.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:19.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:19.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:19.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:19.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:19.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:19.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:19.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:19.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:20.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:20.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:20.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:20.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:21.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:21.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:21.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:21.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:21.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:21.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:21.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:21.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:21.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:21.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:21.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:21.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:21.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:22.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:50:22.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:50:22.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:50:22.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:22.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:22.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:22.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:22.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:22.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:22.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:22.767+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:23.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:23.319+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:23.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:23.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:23.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:23.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:23.329 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:23.327+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:23.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:23.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:50:23.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:23.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:23.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:23.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:23.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:23.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:23.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:23.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:23.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:23.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:23.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:23.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:23.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:23.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:27.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:27.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:27.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:27.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:27.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:27.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:27.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:27.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:27.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:27.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:27.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:27.912 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:50:27.912 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:27.923 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:27.924 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:28.030 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:50:28.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:28.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:28.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.075+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:28.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:28.121 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:28.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:28.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:28.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:29.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:29.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:29.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:29.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:29.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:29.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:29.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:29.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:29.463 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:50:29.463 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:50:29.464 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244629 2023-10-14T00:50:29.465 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:29.465 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:50:29.465 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:50:29.465 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:50:31.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:31.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:31.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:31.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:31.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:31.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:31.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:31.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:31.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:31.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:31.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:31.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:31.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:31.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:32.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:32.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:32.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:32.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:32.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:32.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:35.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:35.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:35.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:35.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:36.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:36.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:36.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:36.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:36.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:36.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:36.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:36.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:37.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:37.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:37.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:37.827+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:37.837 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:37.835+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:37.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:37.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:38.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:38.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:38.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:38.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:38.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:38.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:38.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:38.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:38.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:38.959 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:38.955+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:38.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:38.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:38.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:38.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:38.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:38.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:38.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:38.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:39.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.123+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:39.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:39.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:39.184 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:39.185 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:39.185 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:39.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:39.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:39.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:39.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:39.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:39.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:39.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:39.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:39.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:39.473 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:39.474 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:50:39.475 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:39.484 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:39.607 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:39.607 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:39.629 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.627+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:39.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:39.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:39.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:39.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:39.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:39.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:39.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:39.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:39.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:39.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:40.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:40.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:50:40.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:40.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:40.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:40.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:40.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:40.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:40.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:40.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:40.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:40.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:40.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:40.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:40.979 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:50:40.980 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:50:40.991 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244640 2023-10-14T00:50:40.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:40.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:50:40.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:50:40.992 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:50:41.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:50:41.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:50:41.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:41.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:41.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:41.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:41.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:41.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:41.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:41.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:43.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:43.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:43.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:43.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:43.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:43.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:43.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:43.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:43.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:43.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:43.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:43.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:43.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:43.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:44.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:44.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:44.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:44.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:44.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:44.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:44.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:44.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:45.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:45.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:45.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:45.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:45.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:45.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:46.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:46.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:47.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:50:47.895 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:47.895 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:47.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:47.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:47.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:47.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:47.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:47.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:48.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:48.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:48.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:48.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:48.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:48.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:48.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:48.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:48.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:48.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:49.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:49.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:49.130 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:49.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:49.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:49.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:49.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:49.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:49.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:49.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:50.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:50.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:50.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:50.666 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:50.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:50.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:50.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:50.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.682+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:50.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:50.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:50.725 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:50.726 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:50.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:51.002 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:51.003 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:50:51.016 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:51.029 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:52.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:52.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:52.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:52.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:52.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:52.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:52.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:52.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:52.492 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:50:52.493 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:50:52.493 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244652 2023-10-14T00:50:52.494 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:52.494 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:50:52.494 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:50:52.494 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:50:53.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:53.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:53.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:53.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:53.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:53.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:53.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:53.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:54.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:50:54.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:54.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:54.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:54.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:50:54.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:50:54.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:54.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:54.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:50:54.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:54.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:50:56.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:56.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:50:56.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:56.470+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:50:56.485 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:56.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:50:56.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:56.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:50:56.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:56.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:50:56.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:56.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:50:56.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:50:56.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:50:59.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:59.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:50:59.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:59.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:50:59.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:50:59.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:02.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:02.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:02.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:02.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:02.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:02.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:02.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:02.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:02.495 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:02.496 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:02.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:02.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:02.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:02.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:02.500 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:02.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:02.500 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:51:02.516 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:03.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:03.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:03.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:03.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:03.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:03.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:03.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:03.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:03.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:03.751 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:51:03.751 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:51:03.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244663 2023-10-14T00:51:03.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:03.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:51:03.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:51:03.752 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:51:05.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:05.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:05.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:05.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:05.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:05.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:05.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:05.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:05.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:05.162+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:05.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:05.186+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:05.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:05.210+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:05.228 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:05.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:05.228 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:05.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:05.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:05.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:05.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:05.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:06.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:06.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:06.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:06.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:06.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:06.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:06.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:06.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:06.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:06.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:06.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:06.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:06.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:06.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:08.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:08.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:08.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:08.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:08.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:08.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:08.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:08.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:08.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:08.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:09.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:09.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:09.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:09.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:09.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:09.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:09.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:09.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:09.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:09.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:09.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:09.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:13.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:13.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:13.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:13.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:13.101 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:13.102 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.098+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:13.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:13.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:13.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:13.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:13.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:13.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.366+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:13.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:13.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:13.397 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:13.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:13.406 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:13.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:13.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:13.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:13.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:13.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:13.753 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:51:13.753 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:13.756 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:13.766 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:14.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:13.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:14.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:14.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:14.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:14.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:14.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:14.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:14.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:14.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:14.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:14.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:14.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:14.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:14.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:14.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:14.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:14.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:14.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:14.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:14.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:14.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:15.040 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:51:15.041 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:51:15.043 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244675 2023-10-14T00:51:15.044 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:15.044 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:51:15.044 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:51:15.044 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:51:15.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:15.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:15.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:15.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:15.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:15.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:15.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:15.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:15.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:15.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:15.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:16.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:15.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:17.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:17.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:17.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:17.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:17.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:17.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:17.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:17.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:17.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:17.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:17.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:17.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:19.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:19.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:19.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:19.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:19.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:19.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:19.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:19.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:19.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:19.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:19.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:19.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:19.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:19.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:19.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:19.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:19.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:19.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:19.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:19.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:19.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:19.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:19.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:19.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:19.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:20.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:20.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:20.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:20.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:20.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:20.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:21.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:21.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:21.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:21.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:21.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:21.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:21.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:21.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:21.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:21.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:21.571 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:21.566+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:21.571 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:21.566+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:21.572 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:21.566+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:25.045 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:51:25.045 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:25.046 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:25.072 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:25.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:25.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:25.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:25.234+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:25.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:25.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:25.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:25.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:25.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:25.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:25.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:25.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:25.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:25.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:25.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:25.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:25.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:25.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:25.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:25.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:25.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:51:25.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:51:25.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:25.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:25.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:25.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:25.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:25.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:25.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:25.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:25.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:51:25.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:51:25.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:25.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:25.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:25.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:25.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:25.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:25.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:25.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:25.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:26.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:51:26.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:26.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:26.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:26.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:26.134 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:26.134 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:26.134 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:51:26.319 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:51:26.321 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244686 2023-10-14T00:51:26.321 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:26.321 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:51:26.321 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:51:26.322 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:51:26.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:26.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:26.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:26.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:26.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:26.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:26.594 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.590+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:26.594 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.590+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:26.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:26.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:26.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:26.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:26.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:26.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:26.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:26.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:26.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:27.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:27.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:27.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:27.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:27.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:27.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:27.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:27.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:27.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:27.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:27.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:27.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:27.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:27.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:27.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:27.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:27.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:27.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:27.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:27.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:27.926+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:27.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:27.966+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:27.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:27.978+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:27.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:27.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:27.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:27.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:27.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:27.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:28.377 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:28.374+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:28.414 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:28.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:28.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:28.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:28.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:28.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:28.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:28.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:28.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:28.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:28.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:30.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:30.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:30.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:30.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:30.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:30.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:30.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:30.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:30.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:30.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:30.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:30.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:31.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:51:31.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:31.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:31.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:31.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:31.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:31.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:31.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:31.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:31.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:31.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:31.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:31.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:31.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:31.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:31.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:31.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:31.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:31.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:32.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:32.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:32.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:32.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:32.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:32.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:32.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:32.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:32.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:33.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:33.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:33.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:33.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:33.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:33.256 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:51:33.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:33.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:33.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.297+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:33.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.301+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:33.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.313+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:33.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.313+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:33.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:33.313+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:33.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:33.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:33.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:33.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:33.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:33.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:33.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:34.837 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:34.833+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:34.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:34.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:34.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:34.889+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:34.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:34.901+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:35.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:35.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:35.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:35.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:35.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:35.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:35.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:35.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:35.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:35.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:35.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:35.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:36.333 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:51:36.333 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:36.336 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:36.356 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:36.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:36.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:36.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:36.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:36.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:36.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:36.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:36.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:36.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:36.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:36.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:36.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:36.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:36.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:36.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:51:37.747 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:51:37.753 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244697 2023-10-14T00:51:37.754 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:37.754 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:51:37.754 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:51:37.754 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:51:37.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:37.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:37.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:37.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:37.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:37.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:38.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:38.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:38.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:38.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:38.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:38.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:38.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:38.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:39.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:39.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:39.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:39.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:39.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:39.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:39.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:39.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:41.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:41.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:51:41.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:41.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:51:41.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:41.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:41.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:41.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:41.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:41.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:41.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:41.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:41.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:41.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:41.939 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:41.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:41.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:41.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:41.958 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:41.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:41.958 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:41.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:41.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:41.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:41.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:41.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:42.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:42.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:42.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:42.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:42.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:42.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:42.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:42.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:43.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:43.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:43.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:43.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:43.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:43.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:43.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:43.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:43.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:43.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:43.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:43.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:43.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:43.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:43.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:43.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:43.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:43.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:43.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:43.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:44.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:44.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:44.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:44.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:44.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:44.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:44.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:44.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:44.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:44.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:45.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:45.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:45.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:45.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:45.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:45.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:45.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:47.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:47.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:47.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:47.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:47.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:47.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:47.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:47.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:47.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:47.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:47.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:47.756 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:47.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:47.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:47.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:47.763 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:51:47.790 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:47.790 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:48.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:48.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:48.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:48.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:48.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:48.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:48.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:48.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:48.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:48.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:48.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:48.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:49.021 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:51:49.028 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:51:49.029 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244709 2023-10-14T00:51:49.029 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:49.029 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:51:49.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:51:49.030 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:51:50.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:50.525+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:50.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:50.529+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:50.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:50.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:50.570 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:50.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:50.570 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:50.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:50.570 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:50.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:51.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:51.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:51.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:51.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:51.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:51.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:51.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:51.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:51.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:51.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:51.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:51.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:51.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:51.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:51.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:51.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:51.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:51.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:51.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:51.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:51.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:51.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:51.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:51.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:51.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:51.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:51.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:51.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:52.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:52.185+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:52.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:52.185+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:52.219 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:52.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:52.230 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:52.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:52.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:52.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:52.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:52.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:52.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:52.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:53.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:51:53.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:51:53.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:51:53.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:51:53.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:53.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:53.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:53.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:53.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:53.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:54.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.265+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:51:54.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:54.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:54.286 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:54.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:54.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:54.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:54.373 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:54.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:54.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:54.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:54.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:54.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:54.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:54.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:56.309 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:56.309+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:51:56.309 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:56.309+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:51:56.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:56.329+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:51:56.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:56.329+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:51:56.333 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:56.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:51:56.333 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:56.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:51:56.333 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:51:56.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:51:57.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:57.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:57.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:57.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:57.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:57.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:51:59.051 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:51:59.052 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:59.059 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:59.064 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:59.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:51:59.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:59.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:51:59.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:59.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:51:59.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:59.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:51:59.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:51:59.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:00.454 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:52:00.456 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:52:00.466 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244720 2023-10-14T00:52:00.466 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:00.466 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:52:00.467 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:52:00.467 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:52:00.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:52:00.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:00.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:00.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:00.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:00.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:00.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:00.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:00.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:03.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:03.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:52:03.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:03.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:03.088 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:03.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:03.107 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:03.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:03.130 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:03.129+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:03.130 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:03.129+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:03.130 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:03.129+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:03.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:03.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:03.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:03.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:03.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:03.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:03.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:03.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:03.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:03.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:04.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.313+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:04.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:04.338 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.337+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:04.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.345+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:04.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:04.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:04.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:04.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:04.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:04.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:04.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:04.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:04.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:04.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:04.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:04.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:04.898 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.897+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:04.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.897+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:04.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:04.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:04.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:04.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:04.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:04.958 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.957+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:04.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:04.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:06.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:06.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:06.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:06.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:06.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:06.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:06.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:06.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:06.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:06.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:07.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:07.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:07.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:07.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:07.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:07.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:52:07.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:07.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:07.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:07.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:07.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:07.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:07.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:07.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:07.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:07.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:07.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:07.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:08.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:08.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:08.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:08.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:08.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:08.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:08.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:08.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:08.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:08.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:08.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:08.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:09.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:09.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:09.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:09.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:09.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:09.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:09.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:09.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:09.121 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:09.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:09.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:09.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:10.474 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:10.474 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:52:10.475 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:10.493 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:10.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:10.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:10.564 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:10.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:10.564 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:10.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:11.796 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:52:11.796 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:52:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244731 2023-10-14T00:52:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:11.798 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:52:11.798 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:52:11.798 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:52:13.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:13.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:13.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:13.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:13.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:13.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:13.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:13.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:13.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:13.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:13.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:13.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:15.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:15.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:15.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:15.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:15.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:15.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:15.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:15.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:15.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:15.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:15.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:16.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:16.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:16.095 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:16.093+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:52:16.095 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:16.093+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:16.103 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:16.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:16.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:16.101+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:16.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:16.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:16.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:16.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:16.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:16.117+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:19.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:19.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:19.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:19.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:19.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:19.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:19.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:19.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:19.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:19.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:19.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:19.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:19.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:19.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:20.250 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:20.245+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:20.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:20.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:21.806 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:21.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:21.808 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:52:21.815 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:23.256 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:52:23.259 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:52:23.260 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244743 2023-10-14T00:52:23.260 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:23.261 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:52:23.261 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:52:23.261 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:52:23.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:52:23.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:23.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:23.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:23.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:23.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:23.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:23.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:23.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:23.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:23.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:23.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:23.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:23.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:23.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:23.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:23.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:23.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:23.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:23.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:23.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:24.181 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:24.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:24.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:24.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:24.212 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:24.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:24.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:24.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:24.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:24.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:24.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:24.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:24.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:24.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:24.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:24.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:25.183 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:25.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:25.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:25.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:25.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:25.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:25.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:25.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:25.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:25.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:25.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:25.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:25.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:25.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:25.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:25.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:25.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:25.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:25.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:25.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:25.650 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:25.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:25.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:25.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:26.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:26.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:26.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:26.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:26.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:26.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:27.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:27.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:27.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:27.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:27.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:27.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:27.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:27.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:28.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:28.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:28.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:28.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:28.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:28.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:28.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:28.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:28.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:28.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:28.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:28.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:28.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:28.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:28.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:28.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:28.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:28.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:28.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:28.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:30.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:30.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:30.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:30.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:30.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:30.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:52:30.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:30.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:30.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:30.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:30.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:30.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:30.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:30.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:31.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:31.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:31.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:30.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:31.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:31.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:31.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:31.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:31.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:31.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:31.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:31.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:31.299 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:31.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:31.299 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:31.296+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:31.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:31.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:31.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:31.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:31.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:33.274 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:52:33.274 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:33.281 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:33.297 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:33.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:33.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:52:33.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:33.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:33.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:33.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:33.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:33.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:33.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:33.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:33.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:33.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:34.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:52:34.425 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:52:34.425 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244754 2023-10-14T00:52:34.425 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:34.426 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:52:34.426 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:52:34.426 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:52:35.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:52:35.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:35.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:35.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:35.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:35.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:35.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:35.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:35.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:36.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:36.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:36.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:36.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:36.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:36.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:36.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:36.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:37.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:37.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:37.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:37.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:37.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:37.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:37.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:37.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:38.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:38.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:38.355 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:38.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:38.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:38.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:38.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:38.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:38.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:38.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:38.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:38.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:39.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:52:39.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:39.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:39.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:39.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:39.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:39.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:39.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:39.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:39.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:39.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:39.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:39.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:39.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:39.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:39.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:39.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:40.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:40.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:40.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:40.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:40.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:40.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:40.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:52:40.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:40.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:40.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:40.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:40.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:40.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:42.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:52:42.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:52:42.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:52:42.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:42.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:42.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:42.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:42.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:42.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:42.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:42.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:42.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:42.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:42.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:42.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:42.408 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:42.404+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:42.433 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:42.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:42.433 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:42.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:42.434 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:42.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:44.427 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:52:44.440 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:44.441 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:44.442 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:45.646 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:52:45.648 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:52:45.649 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244765 2023-10-14T00:52:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:52:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:52:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:52:45.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:52:45.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:45.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.796+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:45.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:45.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:45.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:45.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:45.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:45.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:46.202 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:46.200+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:46.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:46.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:50.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:50.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:50.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:50.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:50.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:50.796+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:50.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:50.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:50.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:51.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:51.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:51.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:51.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:51.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:51.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:51.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.420+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:51.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:51.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:51.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:51.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:51.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:51.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:51.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:51.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:51.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:51.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:51.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:51.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:51.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:51.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:51.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:51.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:51.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:52.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:52.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:52:53.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:53.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:53.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:53.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:53.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:53.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:53.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:52:53.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:52:53.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:53.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:53.123 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:53.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:53.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:53.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:53.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:53.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:54.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:54.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:54.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:54.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:54.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:54.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:54.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:54.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:54.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:54.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:52:54.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:54.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:54.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:54.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:52:54.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:54.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:52:54.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:54.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:52:55.022 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:55.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:52:55.022 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:55.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:52:55.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:52:55.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:52:55.652 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:55.653 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:55.656 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:52:55.656 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:56.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:56.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:56.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:56.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:56.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:56.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:56.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:56.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:57.184 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:52:57.187 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:52:57.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244777 2023-10-14T00:52:57.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:57.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:52:57.199 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:52:57.200 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:52:57.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:57.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:57.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:57.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:57.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:57.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:57.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:57.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:57.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:57.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:57.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:57.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:58.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:58.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:58.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:58.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:58.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:52:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:52:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:52:58.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:52:58.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:52:58.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:52:58.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:52:58.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:00.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:00.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:00.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:00.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:00.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:00.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:00.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:00.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:00.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:01.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:01.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:01.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:01.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:01.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:01.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:01.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:01.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:01.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:01.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:03.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:03.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:03.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:03.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:03.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:03.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:04.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:04.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:04.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:04.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:04.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:04.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:04.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:04.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:04.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:04.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:04.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:05.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:05.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:05.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:05.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:05.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:05.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:05.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:05.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:05.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:06.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:53:06.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:53:06.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:06.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:06.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:06.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:06.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:06.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:06.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:06.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:53:06.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:06.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:06.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:06.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:06.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:06.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:06.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:06.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:06.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:06.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:06.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:06.872+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:06.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:06.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:07.206 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:53:07.207 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:07.215 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:07.229 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:07.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:07.716+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:53:07.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:07.716+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:07.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:07.760+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:07.780 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:07.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:07.780 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:07.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:08.460 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:53:08.466 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:53:08.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244788 2023-10-14T00:53:08.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:08.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:53:08.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:53:08.468 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:53:12.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:53:12.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:12.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:12.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:12.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:12.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:12.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:12.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:12.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:12.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:13.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:13.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:13.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:13.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:13.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:13.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:14.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:14.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:14.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:14.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:14.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:14.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:14.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:14.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:16.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:16.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:16.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:16.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:16.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:16.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:16.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:16.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:16.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:16.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:16.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:16.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:17.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:53:17.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:53:17.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:17.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:17.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:17.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:17.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:17.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:17.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:17.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:17.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:18.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:18.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:18.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:18.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:18.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:18.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:18.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:18.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:18.485 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:53:18.486 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:18.486 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:18.493 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:19.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:19.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:19.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:19.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:19.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:19.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:19.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:19.655+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:53:19.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:19.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:19.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:19.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:19.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:19.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:20.165 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:53:20.171 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:53:20.172 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244800 2023-10-14T00:53:20.172 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:20.173 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:53:20.173 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:53:20.173 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:53:20.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:20.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:20.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:20.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:20.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:20.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:20.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:20.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:21.548 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:21.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:21.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:21.583+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:21.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:21.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:21.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:21.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:21.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:21.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:21.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:21.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:21.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:21.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:21.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:21.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:21.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:21.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:21.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:21.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:21.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:21.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:22.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:22.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:22.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:22.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:22.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:22.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:22.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:22.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:22.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:22.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:22.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:22.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:23.428 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:23.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:53:23.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:23.427+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:23.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:23.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:53:23.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:23.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:23.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:23.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:23.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:23.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:23.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:23.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:25.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:25.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:25.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:25.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:26.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:26.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:26.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:26.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:26.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:26.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:26.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:26.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:26.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:26.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:26.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:26.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:27.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:27.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:27.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:27.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:27.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:27.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:53:27.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:53:27.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:27.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:27.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:27.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:27.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:27.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:27.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:27.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:27.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:29.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:29.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:29.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:29.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:29.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:29.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:29.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:29.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:30.179 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:30.183 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:53:30.185 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:30.199 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:31.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:31.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:31.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:31.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:31.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:31.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:31.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:31.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:31.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:31.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:31.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:31.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:31.566 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:53:31.569 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:53:31.570 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244811 2023-10-14T00:53:31.570 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:31.571 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:53:31.571 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:53:31.571 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:53:33.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:33.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:33.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:33.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:33.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:33.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:33.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:33.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:33.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:36.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:36.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:36.986 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:36.983+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:37.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:37.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:37.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:37.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:37.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:37.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:37.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:37.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:37.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:37.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:53:37.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:37.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:37.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:37.791+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:37.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:37.791+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:37.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:37.791+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:38.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:38.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:38.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:38.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:38.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:38.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:38.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:38.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:38.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:38.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:38.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:38.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:39.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:39.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:39.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:39.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:39.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:39.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:39.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:39.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:39.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:39.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:39.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:39.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:39.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:39.984 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:39.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:53:40.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:40.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:40.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:40.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:40.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:40.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:40.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:40.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:40.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:40.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:40.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:40.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:40.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:40.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:40.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:40.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:40.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:40.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:40.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:40.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:41.582 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:41.585 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:53:41.590 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:41.600 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:42.954 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:53:42.954 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:53:42.955 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244822 2023-10-14T00:53:42.955 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:42.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:53:42.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:53:42.956 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:53:44.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:44.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:44.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:44.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:45.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:45.971+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:45.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:45.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:53:45.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:45.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:46.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:46.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:46.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:46.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:46.027 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:46.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:46.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:46.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:46.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:46.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:46.064 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:46.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:46.065 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:46.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:46.065 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:46.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:46.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:46.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:49.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:49.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:49.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:49.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:49.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:49.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:50.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:50.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:53:50.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:50.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:50.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:50.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:50.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:50.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:52.362 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.359+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T00:53:52.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:53:52.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:52.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:53:52.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:52.456 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:52.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:53:52.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:52.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:52.970 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:52.980 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:52.980 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:53:53.012 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:54.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:54.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:53:54.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:54.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:54.198 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:54.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:54.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:54.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:53:54.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:54.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:53:54.318 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:53:54.318 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:53:54.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244834 2023-10-14T00:53:54.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:54.333 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:53:54.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:53:54.334 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:53:56.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:53:56.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:53:56.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:56.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:56.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:56.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:56.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:56.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:56.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:56.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:58.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:58.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:58.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:58.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:53:58.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:58.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:53:58.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:58.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:58.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:58.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:58.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:58.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:58.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:58.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:53:58.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:58.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:53:58.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:58.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:53:58.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:53:58.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:53:59.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:59.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:53:59.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:59.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:53:59.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:59.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:53:59.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:59.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:53:59.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:53:59.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:01.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:01.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:01.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:01.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:01.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:01.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:02.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:02.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:02.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:02.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:02.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:02.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:02.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:02.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:02.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:04.338 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:04.338 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:04.344 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:04.349 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:54:05.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:05.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:05.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:05.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:05.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:05.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:05.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:05.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:05.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:05.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:05.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:05.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:05.595 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:54:05.595 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:54:05.595 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244845 2023-10-14T00:54:05.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:05.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:54:05.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:54:05.596 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:54:07.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:07.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:07.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:07.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:07.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:07.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:07.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:07.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:07.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:07.598+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:07.601 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:07.598+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:07.602 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:07.598+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:08.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:08.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:08.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:08.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:08.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:08.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:08.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:08.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:08.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:08.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:08.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:08.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:08.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:08.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:09.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:09.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:09.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:09.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:09.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:09.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:09.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:09.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:09.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:09.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:09.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:09.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:10.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:10.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:10.401 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:10.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:10.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:10.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:10.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:10.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:10.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:10.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:10.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:10.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:10.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:10.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:10.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:10.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:10.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:10.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:10.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:10.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:10.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:10.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:10.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:10.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:10.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:10.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:10.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:10.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:11.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:11.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:11.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:11.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:11.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:11.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:11.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:11.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:11.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:11.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:11.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:13.035 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:13.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:13.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:13.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:13.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:13.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:13.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:13.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:13.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:13.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:15.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:15.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:15.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:15.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:15.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:15.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:15.599 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:15.617 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:15.617 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:15.618 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:54:15.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:15.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:15.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:15.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:15.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:15.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:15.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:15.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:15.791 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:15.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:15.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:15.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:15.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:15.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:17.036 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:54:17.037 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:54:17.038 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244857 2023-10-14T00:54:17.038 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:17.039 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:54:17.039 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:54:17.039 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:54:17.085 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:17.107 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:17.125 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.122+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:17.228 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:17.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.290+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:17.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:17.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:17.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:17.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:19.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:19.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:19.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:19.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:19.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:19.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:19.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:19.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:19.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:19.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:19.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:19.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:20.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:20.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:20.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:20.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:20.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:20.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:20.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:20.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:20.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:20.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:20.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:20.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:20.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:20.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:20.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:20.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:20.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:20.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:20.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:20.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:20.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:20.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:20.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:20.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:20.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:20.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:21.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:21.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:21.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:21.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:21.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:21.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:22.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:22.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:22.033 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:22.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:22.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:22.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:22.057 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:22.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:22.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:22.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:22.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:22.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:22.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:22.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:22.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:22.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:22.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:22.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:22.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:22.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:22.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:22.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:22.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:25.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:25.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:25.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:25.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:25.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:25.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:54:25.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:25.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:25.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:25.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:25.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:26.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:25.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:26.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:26.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:26.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:26.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:26.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:26.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:26.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:26.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:26.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:26.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:27.043 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:54:27.045 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:27.048 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:27.056 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:27.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:27.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:27.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:27.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:27.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:27.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:27.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:27.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:27.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:27.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:27.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:27.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:27.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:27.886+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:27.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:27.886+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:27.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:27.886+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:28.219 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:28.210+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:28.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:28.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:28.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:28.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:28.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:28.270+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:28.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:28.270+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:28.494 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:54:28.495 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:54:28.495 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244868 2023-10-14T00:54:28.495 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:28.496 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:54:28.496 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:54:28.496 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:54:29.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:29.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:29.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:29.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:29.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:29.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.618+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:29.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:29.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:29.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:29.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:29.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:29.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:29.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:29.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:31.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:31.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:31.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:31.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.162+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:31.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:31.178 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:31.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:31.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:31.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:31.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:31.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:31.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:31.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:31.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:31.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:31.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:31.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:31.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:31.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:31.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:31.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:31.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:31.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:31.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.570+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:31.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.570+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:31.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:31.570+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:32.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:32.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:32.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:32.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:32.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:32.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:33.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:54:33.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:33.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:33.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:33.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:33.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:33.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:33.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:33.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:34.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:54:34.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:34.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:34.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:34.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:34.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:34.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:34.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:34.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:34.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:34.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:34.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:36.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:36.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:36.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:36.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:36.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:36.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:36.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:36.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:36.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:36.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:36.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:36.066+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:36.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:36.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:38.048 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:38.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:38.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:38.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:38.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:38.097 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:38.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:38.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:38.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:54:38.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:38.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:38.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:38.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:38.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:38.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:38.217 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.214+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:38.511 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:54:38.511 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:38.514 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:38.525 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:38.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:54:38.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:38.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:38.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:38.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:38.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:38.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:38.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:38.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:38.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:54:38.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:38.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:38.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:38.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:38.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:38.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:38.824 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:38.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:38.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:38.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:38.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:38.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:38.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:39.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:39.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:39.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:39.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:39.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:39.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:39.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:39.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:39.797 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:54:39.800 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:54:39.801 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244879 2023-10-14T00:54:39.801 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:39.801 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:54:39.801 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:54:39.802 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:54:40.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:40.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:40.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:40.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:40.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:40.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:40.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:40.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:40.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:40.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:40.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:40.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:40.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:40.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:40.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:43.338 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:43.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:43.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:43.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:43.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:43.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:43.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:43.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:43.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:43.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:45.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:45.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:45.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:45.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:45.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:45.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:45.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:45.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:45.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:45.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:45.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:45.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:45.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:45.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:47.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:47.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:47.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:47.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:47.022 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:47.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:47.044 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:47.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:47.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:47.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:47.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:47.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:47.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:47.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:47.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:47.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:47.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:47.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:47.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:47.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:47.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:47.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:47.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:47.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:47.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:47.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:48.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:48.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:54:48.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:48.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:48.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:48.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:54:48.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:48.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:48.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:48.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:48.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:48.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:48.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:48.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:48.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.410+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:48.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:48.449 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:48.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:48.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:48.472 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:48.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:48.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:48.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:48.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:48.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:49.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:49.809 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:49.812 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:54:49.819 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:51.052 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:54:51.054 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:54:51.056 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244891 2023-10-14T00:54:51.057 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:51.057 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:54:51.057 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:54:51.057 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:54:51.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:51.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:51.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:51.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:51.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:51.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:51.844 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:51.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:51.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:51.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:51.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:51.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:51.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:51.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:52.495 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:52.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:52.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:52.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:52.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:52.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:52.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:52.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:53.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:53.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:53.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:53.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:53.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:53.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:53.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:53.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:53.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:53.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:54:57.226 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:57.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:57.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:57.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:57.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:57.338 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:57.858 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:54:57.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.857+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:54:57.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.857+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:54:57.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.857+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:54:57.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.861+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:54:57.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.869+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:54:57.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:54:57.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:54:57.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:54:59.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:59.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:54:59.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:59.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:54:59.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:59.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:54:59.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:59.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:54:59.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:59.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:54:59.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:54:59.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:01.059 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:01.060 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:01.064 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:55:01.078 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:01.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:55:01.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:01.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:01.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:01.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:01.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:01.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:01.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:01.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:01.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:01.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:01.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:01.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:01.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:01.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:02.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:02.173+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:02.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:02.201+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:02.441 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:55:02.441 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:55:02.441 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244902 2023-10-14T00:55:02.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:02.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:55:02.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:55:02.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:55:03.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:03.389+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:03.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:03.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:55:03.426 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:03.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:03.447 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:03.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:03.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:03.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:03.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:03.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:04.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:04.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:04.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:04.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:04.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:04.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:04.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:04.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:04.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:04.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:05.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:05.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:05.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:05.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:05.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:05.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:05.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:05.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:05.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:05.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:05.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:05.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:05.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:05.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:05.152 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:05.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:55:05.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:05.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:05.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:05.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:05.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:05.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:07.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:07.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:07.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:07.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:07.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:07.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:07.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:07.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:08.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:08.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:08.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:08.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:08.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:08.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:08.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:08.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:08.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:08.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:10.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:10.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:10.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:10.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:10.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:10.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:10.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:10.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:10.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:10.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:11.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:55:11.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:11.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:11.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:11.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:11.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:11.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:11.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:11.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:11.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:11.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:11.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:11.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:11.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:11.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:11.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:12.442 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:55:12.450 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:12.451 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:12.455 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:12.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:12.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:12.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:12.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:12.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:12.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:12.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:12.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:12.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:12.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:12.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:12.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:12.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:12.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:13.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:13.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:13.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:13.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:13.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:13.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:13.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:13.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:13.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:13.256 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:13.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:13.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:13.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:13.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:13.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:13.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:13.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:13.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:13.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:13.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:13.362 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:55:13.357+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:55:13.907 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:55:13.908 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:55:13.925 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244913 2023-10-14T00:55:13.925 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:13.926 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:55:13.926 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:55:13.926 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:55:14.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:14.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:14.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:55:14.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:14.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:14.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:14.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:14.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:14.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:14.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:14.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:14.455 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.449+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:14.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:55:14.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:14.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.677+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:55:14.687 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:14.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.689+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:14.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.689+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:14.694 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:14.689+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:14.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:55:14.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:55:14.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:14.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:15.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:15.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:15.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:15.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:15.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:15.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:15.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:15.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:15.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:15.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:15.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:15.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:15.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:15.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:16.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:16.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:16.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:16.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:16.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:16.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:16.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:16.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:16.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:16.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:17.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:17.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:17.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:17.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:17.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:17.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:17.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:17.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:17.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:17.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:17.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:17.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:17.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:17.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:18.499 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:18.497+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:18.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:18.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:55:18.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:18.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:18.572 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:18.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:18.573 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:18.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:18.573 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:18.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:19.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:19.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:19.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:19.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:19.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:19.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:19.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:19.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:20.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:20.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:20.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:20.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:20.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:20.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:20.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:20.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:20.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:20.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:20.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:20.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:20.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:20.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:21.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:21.277+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:55:21.290 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:21.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:21.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:21.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:21.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:21.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:21.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:21.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:21.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:21.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:21.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:21.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:22.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:22.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:22.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:22.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:22.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:22.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:22.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:22.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:22.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:22.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:22.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:22.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:22.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:22.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:22.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:22.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:55:22.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:22.641+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:22.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:22.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:22.675 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:22.673+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:22.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:22.745+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:22.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:22.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:23.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:23.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:23.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:23.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:23.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:23.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:23.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:23.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:23.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:23.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:23.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:23.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:23.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:23.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:23.929 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:23.929 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:55:23.935 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:23.936 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:24.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:24.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:24.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:24.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:25.233 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:55:25.233 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:55:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244925 2023-10-14T00:55:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:55:25.236 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:55:25.236 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:55:26.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:26.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:26.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:26.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:26.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:26.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:26.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:26.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:26.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:26.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:26.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:26.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:26.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:26.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:27.895 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:55:27.893+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:55:29.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:29.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:29.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:29.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:30.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:30.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:30.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:30.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:30.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:30.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:55:30.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:30.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:30.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:30.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:34.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:34.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:34.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:34.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:35.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:35.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:35.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:35.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:35.258 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:55:35.258 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:35.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:35.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:35.272 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:35.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:35.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:35.279 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:35.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:35.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:36.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:36.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:36.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:36.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:36.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:36.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:36.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:36.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:36.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:36.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:36.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:36.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:36.641 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:55:36.646 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:55:36.647 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244936 2023-10-14T00:55:36.647 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:36.647 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:55:36.647 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:55:36.648 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:55:38.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:38.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:38.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:38.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:38.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:38.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:40.126 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:40.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:40.150 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:40.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:55:40.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:40.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:40.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:40.177+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:40.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:40.177+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:40.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:40.177+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:42.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:42.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:42.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:42.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:42.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:42.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:42.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:42.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:42.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:42.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:42.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:42.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:42.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:42.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:44.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:44.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:44.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:44.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:44.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:44.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:44.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:44.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:45.235 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:45.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:46.662 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:55:46.671 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:46.676 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:46.677 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:47.973 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:55:47.973 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:55:47.974 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244947 2023-10-14T00:55:47.974 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:47.974 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:55:47.974 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:55:47.975 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:55:48.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:48.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:48.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:48.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:48.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:48.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:48.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:48.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:48.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:48.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:48.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:48.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:48.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:48.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:48.536 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:55:48.532+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:55:51.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:51.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:51.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:51.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:51.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:51.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:51.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:51.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:51.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:51.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:51.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:51.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:51.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:51.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:51.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:51.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:53.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:53.128+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:53.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:53.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:55:53.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:53.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:55:53.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:53.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:55:53.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:53.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:53.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:53.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:53.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:53.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:53.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:53.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:53.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:53.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:53.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:53.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:54.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:54.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:54.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:54.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:54.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:54.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:54.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:54.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:54.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:54.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:54.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:54.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:54.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:54.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:55:54.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:55:54.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:55:55.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:55.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:55.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:55.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:55.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:55.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:55.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:55.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:55.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:55.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:55.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:55.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:55.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:55.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:56.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:56.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:56.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:56.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:56.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:56.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:57.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:57.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:57.063 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:57.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:57.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:57.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:57.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:57.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:57.998 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:55:57.998 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:58.008 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:58.014 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:59.258 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:55:59.258 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:55:59.259 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244959 2023-10-14T00:55:59.259 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:59.259 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:55:59.260 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:55:59.260 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:55:59.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:59.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:59.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:59.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:59.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:59.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:55:59.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:55:59.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:55:59.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:55:59.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:55:59.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:55:59.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:55:59.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:55:59.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:55:59.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:00.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:00.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:00.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:00.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:00.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:00.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:00.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:00.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:00.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:01.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:01.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:01.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:01.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:01.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:01.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:01.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:01.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:01.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:01.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:03.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:03.708+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:03.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:03.708+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:03.723 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:03.720+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:04.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:04.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:04.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:04.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:04.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:04.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:04.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:04.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:04.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:04.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:04.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:04.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:06.106 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:06.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:06.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:06.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:06.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:06.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:06.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:06.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:06.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:06.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:06.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:06.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:06.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:06.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:06.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:06.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:06.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:06.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:06.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:06.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:06.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:08.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:08.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:08.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:08.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:08.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:08.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:09.274 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:56:09.274 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:09.292 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:09.302 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:10.797 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:56:10.797 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:56:10.798 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244970 2023-10-14T00:56:10.798 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:10.798 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:56:10.799 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:56:10.799 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:56:10.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:10.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:10.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:10.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:10.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:10.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:10.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:10.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:10.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:10.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:10.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:10.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:10.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:10.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:11.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:11.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:11.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:11.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:11.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:11.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:11.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:11.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:11.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:11.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:14.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:14.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:14.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:14.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:14.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:14.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:14.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:14.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:14.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:14.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:14.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.688+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:14.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:14.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:14.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:14.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:14.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:14.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:14.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:14.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:14.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:14.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:14.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:14.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:15.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:15.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:15.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:15.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:15.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:15.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:15.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:15.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:15.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:15.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:15.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:16.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:16.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:16.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:16.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:16.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:16.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:16.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:16.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:16.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:16.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:16.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:16.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:19.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:19.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:19.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:19.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:19.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:19.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:19.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:19.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:19.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:19.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:19.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:19.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:20.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:20.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:20.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:20.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:20.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:20.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:20.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:20.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:20.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:20.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:20.800 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:20.802 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:20.807 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:20.808 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:56:21.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:21.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:21.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:21.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:21.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:21.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:21.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:21.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:21.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:21.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:22.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:22.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:22.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:22.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:22.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:22.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:22.435 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:56:22.440 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:56:22.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244982 2023-10-14T00:56:22.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:22.442 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:56:22.443 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:56:22.443 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:56:23.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:23.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:23.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:23.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:23.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:23.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:23.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:23.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:23.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:23.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:23.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:23.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:23.840+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:23.850 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:23.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:23.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:23.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:23.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:23.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:23.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:23.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:26.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:26.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:26.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:26.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:26.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:26.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:26.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:26.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:26.798 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:26.796+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:26.812 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:26.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:26.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:26.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:26.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:26.824+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:26.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:26.824+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:26.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:26.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:26.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:26.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:27.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:56:27.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:27.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:27.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:27.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:27.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:27.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:27.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:27.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:27.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:27.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:27.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:27.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:27.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:27.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:27.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:28.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:28.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:28.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:28.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:28.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:28.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:28.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:28.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:29.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:29.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:29.409 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:29.408+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:29.425 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:29.424+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:29.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:29.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:29.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:29.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:30.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:30.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:30.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:30.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:30.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:30.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:30.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:30.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:30.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:30.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:30.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:30.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:30.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:30.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:30.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:31.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:31.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:31.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:31.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:31.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:31.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:31.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:31.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:32.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:32.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:32.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:32.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:32.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:32.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:32.446 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:56:32.459 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:32.460 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:32.468 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:32.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:32.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:32.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:32.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:32.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:32.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:33.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:33.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:33.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:33.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:33.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:33.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:33.851 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:56:33.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:33.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:33.864 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:56:33.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:33.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:33.870 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697244993 2023-10-14T00:56:33.870 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:33.870 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:56:33.870 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:56:33.871 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:56:34.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:56:34.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:34.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:34.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:34.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:34.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:34.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:34.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:34.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:34.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:34.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:34.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:34.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:36.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:36.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:36.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:36.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:56:36.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:56:36.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:36.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:36.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:36.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:36.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:36.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:36.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:36.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:37.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:37.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:37.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:37.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:37.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:37.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:37.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:37.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:37.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:37.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:39.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:39.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:39.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:39.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:39.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:39.915+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:40.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:40.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:40.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:40.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:40.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:40.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:40.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:40.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:41.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:41.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:41.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:41.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:41.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:41.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:41.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:41.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:41.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:41.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:43.892 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:56:43.892 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:43.893 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:43.902 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:44.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:44.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:44.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:44.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:44.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:44.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:44.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:44.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:44.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:44.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:44.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:44.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:44.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:44.511+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:44.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:44.515+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:44.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:44.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:44.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:44.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:44.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:44.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:44.525 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:44.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:56:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:56:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697245005 2023-10-14T00:56:45.112 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:45.112 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:56:45.112 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:56:45.112 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:56:45.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:45.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:45.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:45.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:45.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:45.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:45.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:45.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:45.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:45.599 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:45.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:45.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:45.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:45.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:45.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:45.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:45.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:45.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:45.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:45.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:45.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:45.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:45.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:45.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:45.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:46.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:46.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:46.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:46.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:46.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:46.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:47.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:47.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:47.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:47.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:47.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:47.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:47.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:47.143+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:47.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:47.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:47.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:47.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:47.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:47.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:47.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:47.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:47.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:47.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:47.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:47.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:47.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:47.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:47.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:49.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:49.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:49.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:49.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:49.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:49.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:49.504 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:49.499+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:49.510 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:49.507+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:49.510 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:49.507+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:50.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:50.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:50.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:50.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:50.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:50.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:50.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:50.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:50.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:51.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:50.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:51.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:51.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:51.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:51.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:51.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:51.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:51.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:51.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:51.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:51.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:51.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:51.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:51.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:51.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:52.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:52.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:52.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:52.187+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:52.216 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:52.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:52.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:52.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:52.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:52.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:52.218 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:52.215+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:53.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:53.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:56:53.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:53.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:56:53.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:53.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:53.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:53.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:53.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:53.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:53.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:53.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:53.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:53.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:54.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:54.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:54.608 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:56:54.628 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.627+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:54.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:56:54.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:56:54.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:56:54.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:54.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:54.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:54.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:54.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:54.746 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:54.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:54.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:55.155 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:55.156 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:55.160 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:56:55.173 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:55.355 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:55.347+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:55.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:55.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:56:56.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:56.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:56:56.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:56.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:56:56.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:56.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:56.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:56.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:56.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:56.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:56.394 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:56:56.399 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:56:56.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697245016 2023-10-14T00:56:56.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:56.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:56:56.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:56:56.401 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:56:58.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:58.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:58.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:58.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:59.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:59.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:56:59.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:59.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:56:59.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:56:59.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:56:59.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:59.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:56:59.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:59.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:56:59.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:56:59.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:01.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:01.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:01.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:01.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:01.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:01.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:02.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:02.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:02.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:02.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:02.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:02.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:02.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:02.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:02.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:02.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:02.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:02.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:02.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:02.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:03.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:03.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:03.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:03.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:03.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:03.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:03.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:03.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:03.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:03.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:03.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:03.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:03.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:03.791+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:03.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:03.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:03.834 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:03.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:03.844 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:03.843+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:03.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:03.843+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:06.417 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:06.426 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:57:06.436 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:06.447 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:07.815 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:57:07.815 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:57:07.815 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697245027 2023-10-14T00:57:07.816 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:07.816 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:57:07.816 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:57:07.816 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:57:08.213 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:08.211+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:57:08.213 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:08.211+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:08.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:08.243+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:08.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:08.259+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:08.280 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:08.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:08.280 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:08.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:08.280 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:08.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:10.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:10.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:57:10.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:10.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:10.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:10.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:11.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:11.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:11.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:11.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:11.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:11.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:11.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:11.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:11.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:11.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:11.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:11.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:11.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:11.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:13.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:57:13.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:13.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.559+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:13.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:13.594 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:13.594 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:13.594 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:13.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:13.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:13.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:13.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:13.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:13.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:13.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:13.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:13.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:13.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:16.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:16.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:16.026 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:16.023+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:16.064 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:16.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:16.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:16.083+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:16.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:16.083+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:16.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:16.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:16.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:16.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:16.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:16.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:16.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:16.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:16.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:16.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:17.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:57:17.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:57:17.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:17.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:17.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:17.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:17.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:17.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:17.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:17.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:17.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:17.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:17.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:17.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:17.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:17.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:17.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:17.822 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:17.825 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:17.831 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:57:17.839 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:18.998 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:57:19.005 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:57:19.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697245039 2023-10-14T00:57:19.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:19.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:57:19.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:57:19.008 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:57:21.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:21.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:21.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:21.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:21.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:21.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:21.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:21.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:21.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:21.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:21.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:21.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:22.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:22.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:22.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:22.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:22.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:22.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:22.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:22.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:22.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:22.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:23.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:23.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:23.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:23.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:23.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:23.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:23.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:23.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:23.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:23.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:23.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:23.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:23.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:23.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:26.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:26.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:26.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:26.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:26.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:26.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:26.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:26.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:26.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:26.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:27.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:27.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:27.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:27.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:27.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:27.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:27.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:27.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:29.037 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:29.040 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:29.043 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:57:29.043 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:29.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:29.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:29.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:29.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:29.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:29.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:29.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:29.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:29.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:29.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:29.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:29.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:29.692 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:29.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:57:29.692 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:29.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:29.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:29.714+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:29.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:29.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:29.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:29.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:29.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:29.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:29.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:29.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:30.123 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:30.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:30.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.166+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:30.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:30.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:30.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:30.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:30.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:30.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:30.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:30.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:30.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:30.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:30.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:30.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:30.446 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:57:30.447 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:57:30.447 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697245050 2023-10-14T00:57:30.448 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:30.448 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ -n 172.21.15.138:0/393253116 ]] 2023-10-14T00:57:30.448 INFO:tasks.workunit.client.0.smithi138.stderr:+ [[ 172.21.15.138:0/393253116 != \1\7\2\.\2\1\.\1\5\.\1\3\8\:\0\/\3\9\3\2\5\3\1\1\6 ]] 2023-10-14T00:57:30.448 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:57:30.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:30.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:30.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:30.802 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:30.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:30.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:30.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:31.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:31.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:31.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:31.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:31.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:31.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:31.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:31.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:31.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:31.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:31.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:31.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:31.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:31.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:31.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:31.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:33.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:33.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:33.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:33.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:33.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:33.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:33.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:57:33.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:57:33.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:33.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:33.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:33.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:33.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:33.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:33.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:33.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:33.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:33.873 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:33.870+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:57:33.873 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:33.870+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:33.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:33.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:33.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:33.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:33.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:33.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:33.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:33.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:33.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:33.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:34.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:34.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:34.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:34.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:34.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:34.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:34.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:34.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:34.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:34.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:34.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:34.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:35.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:35.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:35.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:35.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:35.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:35.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:35.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:35.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:35.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:35.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:35.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:35.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:37.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:37.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:37.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:37.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:37.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:37.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:37.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:37.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:37.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:37.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:38.591 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:38.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:38.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:38.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:39.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:39.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:39.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:39.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:39.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:39.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:39.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:39.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:39.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:40.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:40.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.026+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:40.045 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:40.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:40.090 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:40.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:40.396 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.394+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:40.424 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.422+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:40.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.426+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:40.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:40.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:40.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:40.463 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:40.483 INFO:tasks.workunit.client.0.smithi138.stderr:++ ceph mgr dump 2023-10-14T00:57:40.485 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:40.486 INFO:tasks.workunit.client.0.smithi138.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:41.961 INFO:tasks.workunit.client.0.smithi138.stderr:+ CLIENT_ADDR=172.21.15.138:0/393253116 2023-10-14T00:57:41.961 INFO:tasks.workunit.client.0.smithi138.stderr:++ date +%s 2023-10-14T00:57:41.962 INFO:tasks.workunit.client.0.smithi138.stderr:+ CURRENT_TIME=1697245061 2023-10-14T00:57:41.962 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i = 1 )) 2023-10-14T00:57:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:57:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:57:42.222 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:57:42.218+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:57:42.223 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:57:42.232 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:57:42.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:42.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:42.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:42.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:42.466 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:42.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:42.467 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:42.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:42.467 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:42.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:42.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:42.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:42.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:42.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:42.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:42.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:42.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:42.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:42.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:42.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:42.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:42.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:43.066 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:43.107 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:43.107 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:43.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:43.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.966+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:43.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:43.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:43.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:43.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:44.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:44.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:44.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:44.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:44.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:44.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:44.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:44.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:57:44.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:44.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:57:44.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:44.962+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:57:44.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:44.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:57:44.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:44.994+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:57:44.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:44.994+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:57:44.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:57:44.994+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:57:45.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:45.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:45.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:45.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:45.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:45.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:45.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:45.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:45.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:45.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:45.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:45.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:45.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:48.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:48.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:48.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:48.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:48.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:48.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:48.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:48.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:48.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:48.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:48.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:48.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:48.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:48.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:48.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:48.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:48.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:48.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:48.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:48.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:50.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:50.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:57:50.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:50.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:50.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:50.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:50.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:50.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:50.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:50.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:50.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:50.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:51.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:51.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:51.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:51.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:51.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:51.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:51.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:51.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:51.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:51.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:51.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:51.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:51.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:51.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:52.236 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:57:52.237 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:57:52.237 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:57:52.407 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:57:52.402+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:57:52.412 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:57:52.426 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:57:53.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:53.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:53.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:53.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:53.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:53.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:53.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:53.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:53.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:53.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:53.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:53.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:56.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:56.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:56.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:56.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:56.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:56.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:56.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:56.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:56.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:57:56.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:56.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:56.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:56.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:56.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:56.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:57.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:57.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:57.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:57.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:57.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:57.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:57.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:57.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:57.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:57.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:57.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:57.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:57.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:59.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:59.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:57:59.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:57:59.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:57:59.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:57:59.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:57:59.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:57:59.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:57:59.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:57:59.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:00.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:00.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:00.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:00.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:00.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:00.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:00.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:00.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:00.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:01.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:01.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:02.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:58:02.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:02.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:02.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:02.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:02.430 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:58:02.431 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:58:02.431 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:58:02.597 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:58:02.594+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:58:02.598 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:58:02.617 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:58:02.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:02.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:03.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:03.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:03.132 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:03.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:03.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:03.162+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:03.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:03.162+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:03.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:03.162+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:04.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:04.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:04.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:04.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:04.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:04.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:04.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:04.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:04.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:05.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:05.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:05.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:05.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:05.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:05.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:05.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:05.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:05.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:05.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:05.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:05.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:06.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:06.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:06.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:06.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:06.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:06.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:06.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:06.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:06.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:06.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:08.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:08.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:58:08.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:08.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:08.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:08.646+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:08.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:08.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:08.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:08.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:08.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:08.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:08.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:08.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:10.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:10.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:10.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:10.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:10.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:10.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:11.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:11.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:11.564 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:11.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:11.568 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:11.566+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:11.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:11.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:11.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:11.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:11.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:11.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:11.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:11.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:11.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:11.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:11.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:11.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:11.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:11.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:11.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:11.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:12.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:58:12.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:12.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:12.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:12.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:12.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:12.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:12.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:12.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:12.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:58:12.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:58:12.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:12.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:12.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:12.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:12.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:12.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:12.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:12.619 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:58:12.620 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:58:12.620 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:58:12.775 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:58:12.770+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:58:12.775 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:58:12.790 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:58:13.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:13.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:13.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:13.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:13.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:13.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:13.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:13.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:13.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:13.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:13.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:13.512 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:13.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:13.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:13.514+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:13.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:13.518+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:13.521 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:13.518+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:14.347 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:14.342+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:14.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:14.366+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:14.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:14.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:14.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:14.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:15.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:15.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:15.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:15.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:58:15.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:58:15.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:15.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:15.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:15.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:15.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:15.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:15.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:15.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:15.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:15.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:15.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:15.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:15.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:58:15.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:15.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:15.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:15.528 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:15.528 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:15.528 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:15.695 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:15.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:15.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:15.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:15.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:16.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:16.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:58:16.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:16.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:16.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:16.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:16.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:16.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:16.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:16.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:16.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:16.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:16.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:16.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:16.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:16.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:19.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:19.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:19.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:19.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:19.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:19.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:19.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:19.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:20.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:20.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:20.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:20.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:20.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:20.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:22.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:22.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:22.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:22.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:22.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:22.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:22.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:22.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:22.795 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:58:22.796 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:58:22.796 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:58:22.915 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:58:22.909+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:58:22.916 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:58:22.937 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:58:22.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:22.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:22.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:22.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:23.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:23.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:23.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:23.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:25.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.625+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:25.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:25.659 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:25.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:25.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.841+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:58:25.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.841+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:25.877 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:25.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.885+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:25.907 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.901+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:25.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:25.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:25.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:25.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:25.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:25.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:25.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:25.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:25.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:25.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:25.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:25.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:25.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:25.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:25.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:27.049 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:27.045+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:58:27.076 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:27.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:27.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:27.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:27.091 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:27.085+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:27.094 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:27.089+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:27.098 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:27.093+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:27.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:27.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:28.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:28.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:28.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:28.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:28.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:28.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:28.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:28.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:29.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:29.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:29.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:29.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:29.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:29.777+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:29.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:29.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:30.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:30.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:30.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:30.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:30.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:30.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:30.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:30.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:30.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:30.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:30.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:30.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:32.939 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:58:32.939 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:58:32.939 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:58:33.094 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:58:33.089+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:58:33.095 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:58:33.099 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:58:33.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:33.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:33.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:33.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:33.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:33.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:33.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:33.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:33.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:33.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:33.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:33.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:33.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:33.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:33.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:33.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:33.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:33.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:33.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:33.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:34.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:34.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:35.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:34.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:35.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:35.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:35.009 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:35.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:35.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:35.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:35.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:35.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:35.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:35.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:35.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:35.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:35.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:35.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:35.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:35.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:35.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:35.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:35.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:36.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:36.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:36.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:36.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:36.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:36.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:36.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:36.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:36.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:36.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:37.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.157+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:37.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:37.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.173+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:37.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.173+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:37.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.173+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:37.819 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.813+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:37.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:37.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:37.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:37.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:38.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:38.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:38.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:38.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:38.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:38.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:38.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:38.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:38.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:38.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:38.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:38.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:40.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:40.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:40.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:40.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:40.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:40.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:40.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:40.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:40.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:40.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:40.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T00:58:40.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:58:40.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:40.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:40.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:40.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:40.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:40.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:40.849+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:42.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:42.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:42.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:42.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:42.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:42.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:42.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:42.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:58:42.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:42.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:42.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:42.325+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:42.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:42.325+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:42.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:42.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:42.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:42.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:42.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:42.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:42.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:42.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:42.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:42.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:42.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:42.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:42.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:42.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:42.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:42.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:42.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:42.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:43.101 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:58:43.101 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:58:43.102 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:58:43.232 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:58:43.233+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:58:43.233 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:58:43.242 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:58:44.643 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T00:58:44.641+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T00:58:44.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:58:44.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:44.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:44.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:44.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:44.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:44.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:44.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:44.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:44.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:44.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:44.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:44.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:44.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:44.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:44.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:47.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:47.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:47.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:47.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:47.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:47.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:48.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:48.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:48.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:48.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:48.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:48.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:48.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:48.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:48.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:48.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:48.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:48.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:48.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:48.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:49.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:49.193+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:58:49.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:49.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:49.268 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:49.265+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:49.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:49.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:49.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:49.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:49.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:49.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:51.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:51.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:58:51.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:51.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:51.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:51.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:51.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:51.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:51.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:51.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:51.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:51.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:51.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:51.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:52.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:52.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:52.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:52.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:52.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:52.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:52.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:52.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:52.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:52.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:52.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:52.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:53.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:53.189+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:53.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:53.205+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:53.226 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:53.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:53.226 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:53.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:53.227 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:53.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:53.243 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:58:53.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:58:53.244 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:58:53.432 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:58:53.429+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:58:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:58:53.446 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:58:53.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:53.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:53.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:53.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:53.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:53.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:53.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:53.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:53.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:53.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:53.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:53.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:53.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:53.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:54.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:54.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:54.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:54.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:54.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:54.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:54.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:54.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:54.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:54.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:54.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:54.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:58:57.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:57.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:58:57.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:57.329+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:58:57.347 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:57.345+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:58:57.347 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:57.345+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:58:57.348 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:58:57.345+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:58:59.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:59.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:58:59.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:59.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:58:59.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:59.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:58:59.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:59.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:58:59.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:59.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:58:59.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:59.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:58:59.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:58:59.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:01.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:01.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:59:01.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:01.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:02.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:02.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:02.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:02.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:02.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.061+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:02.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:02.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:02.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:02.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:02.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:02.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:02.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:02.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:03.447 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:59:03.447 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:59:03.447 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:59:03.546 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:59:03.545+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:59:03.552 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:59:03.574 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:59:03.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:03.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:03.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:03.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:03.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:03.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:03.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:03.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:03.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:03.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:03.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:03.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:04.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:04.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:04.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:04.288 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:04.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:04.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:04.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:04.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:04.925+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:05.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:05.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:05.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:05.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:05.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:05.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:05.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:05.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:05.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:05.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:05.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:05.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:05.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:08.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:08.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:08.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:08.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:08.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:08.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:08.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:08.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:08.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:08.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:08.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:08.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:08.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:08.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:08.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:08.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:09.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:09.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:09.842 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.840+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:09.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:09.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:09.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:09.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:09.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:09.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:09.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:09.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:09.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:09.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:09.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:09.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:09.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:09.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:10.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:10.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:10.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:10.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:10.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:10.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:10.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:10.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:10.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:10.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:10.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:10.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:10.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:10.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:10.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:10.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:11.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:11.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:11.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:11.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:11.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:11.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:11.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:11.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:11.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:11.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:11.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:11.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:11.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:11.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:11.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:11.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:11.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:11.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:11.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:11.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:11.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:11.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:11.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:11.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:11.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:11.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:11.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:11.882 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:11.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:13.576 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:59:13.577 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:59:13.577 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:59:13.711 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:59:13.708+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:59:13.712 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:59:13.721 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:59:15.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:15.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:15.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:15.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:15.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:15.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:15.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:15.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:15.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:15.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:15.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:15.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:15.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:15.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:16.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:16.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:16.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:16.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:16.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:16.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:16.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:16.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:16.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:16.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:16.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:16.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:16.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:16.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:17.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:17.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:17.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:17.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:17.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:17.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:17.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:17.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:17.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:17.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:17.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:17.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:17.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:17.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:19.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:19.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:19.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:19.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:19.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:19.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:19.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:19.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:19.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:19.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:19.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:20.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:20.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:20.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:20.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:20.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:20.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:20.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:20.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:20.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:20.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:20.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:20.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:20.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:20.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:20.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:20.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:20.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:20.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:20.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:20.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:20.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:20.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:21.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:21.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:21.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:21.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:21.430 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:21.428+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:21.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:21.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:21.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:21.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:21.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:21.436+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:23.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:23.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:23.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:23.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:23.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:23.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:23.722 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:59:23.722 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:59:23.722 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:59:23.864 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:59:23.864+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:59:23.869 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:59:23.877 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:59:24.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:24.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:24.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:24.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:24.015 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:24.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:24.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:24.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:24.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:24.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:24.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:24.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:24.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:24.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:24.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:24.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:24.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:24.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:24.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:24.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:24.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:24.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:24.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:24.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:24.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:24.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:27.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:27.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:27.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:27.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:27.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:27.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:27.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:27.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:27.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:27.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:27.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:28.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:28.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:28.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:28.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:28.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:28.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:28.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:28.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:28.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:28.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:28.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:28.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:29.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:29.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:29.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:29.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:30.229 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:30.228+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:30.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:30.252+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:30.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:30.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:30.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:30.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:30.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:30.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:31.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:31.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:31.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:31.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:31.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:31.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:31.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.476+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:31.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:31.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:32.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:32.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:32.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:32.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:32.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:32.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:32.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:32.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:33.879 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:59:33.879 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:59:33.879 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:59:33.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:33.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:33.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:33.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:33.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:33.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:33.949 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:33.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:33.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:33.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:33.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:33.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:33.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:33.956+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:33.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:33.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T00:59:33.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:33.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:59:33.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:33.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:34.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:34.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:34.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:34.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:34.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:34.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:34.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:34.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:34.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:34.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:34.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:34.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:34.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:34.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:34.096 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:59:34.092+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:59:34.096 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:59:34.124 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:59:34.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:34.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:34.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:34.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:34.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:34.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:36.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:36.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:36.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:36.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:36.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:36.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:36.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:36.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:36.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:36.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:36.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:36.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:36.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.952+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:36.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.952+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:36.976 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:36.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:36.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:36.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:36.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:38.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:38.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:38.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:38.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:38.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:38.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:38.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:38.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:38.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:38.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:38.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:38.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:39.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:39.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:39.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:39.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:39.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:39.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:39.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:39.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:40.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:40.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:40.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:40.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:40.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:40.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:40.018 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:40.016+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:40.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:40.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:40.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:40.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:40.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:40.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:40.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:40.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:40.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:40.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:40.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:40.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:42.253 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:42.252+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:42.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:42.256+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:42.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:42.280+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:42.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:42.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:42.323 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:42.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:42.323 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:42.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:42.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:42.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:42.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:42.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:42.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:42.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:42.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:42.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:42.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:42.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:42.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:42.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:42.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:42.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:59:43.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:43.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:43.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:43.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:43.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:43.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:43.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:43.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:43.040 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:43.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:43.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:43.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:43.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:43.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:44.140 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:59:44.141 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:59:44.141 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:59:44.358 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:59:44.356+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:59:44.360 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:59:44.405 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:59:45.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T00:59:45.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T00:59:45.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:45.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:45.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:45.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:45.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:45.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:45.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:45.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:46.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:46.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:46.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:46.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:46.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:46.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:46.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:46.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:46.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:46.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:47.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:47.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T00:59:47.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:47.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:47.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:47.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T00:59:47.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:47.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:47.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:47.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:47.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:47.808+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T00:59:54.411 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T00:59:54.411 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T00:59:54.411 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T00:59:54.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:54.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:54.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:54.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:54.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:54.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:54.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:54.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:54.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:54.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:54.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:54.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:54.507 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T00:59:54.504+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T00:59:54.508 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T00:59:54.527 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T00:59:54.564 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:54.564+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T00:59:54.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:54.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:55.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:55.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:55.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:55.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:55.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:55.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:55.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:55.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:55.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:55.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:55.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:55.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:55.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:56.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:56.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:56.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:56.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:56.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:56.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:56.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:56.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:56.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:56.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:56.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:56.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:58.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:58.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:58.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:58.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:58.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:58.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T00:59:58.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T00:59:58.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T00:59:58.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T00:59:58.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T00:59:58.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T00:59:58.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T00:59:58.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T00:59:59.277 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:59.275+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T00:59:59.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:59.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T00:59:59.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:59.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T00:59:59.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T00:59:59.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:00.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:00.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:00.780 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:00.779+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:00.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:00.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:00.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:00.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:00.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:00.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:00.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:00.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:01.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:01.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:02.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:02.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:02.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:02.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:02.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:02.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:04.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:04.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:04.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:04.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:04.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:04.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:04.528 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:00:04.528 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:00:04.528 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:00:04.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:04.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:04.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:04.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:04.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:04.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:04.708 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:00:04.703+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:00:04.711 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:00:04.732 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:00:06.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:06.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:06.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:06.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:06.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:06.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:06.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:06.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:06.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:06.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:06.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:06.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:10.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:10.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:10.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:10.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:10.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:10.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:00:10.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:10.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:10.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:10.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:10.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:11.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:00:11.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:00:11.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:11.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:11.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:11.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:11.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:11.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:11.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:11.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:11.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:11.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:11.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:11.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:11.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:11.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:11.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:11.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:11.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:12.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:12.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:12.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:12.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:12.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:12.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:12.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:12.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:12.266 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:12.263+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:12.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:12.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:12.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:12.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:12.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:12.279+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:12.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:12.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:12.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:12.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:13.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:13.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:13.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:13.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:13.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:13.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:13.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:13.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:13.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:13.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:13.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:13.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:13.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:13.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:14.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:00:14.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:14.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:14.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:14.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:14.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:14.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:14.677 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:14.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.687+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:14.690 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.687+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:14.691 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:14.687+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:14.735 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:00:14.735 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:00:14.736 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:00:14.885 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:00:14.883+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:00:14.885 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:00:14.899 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:00:15.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:15.571+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:00:15.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:15.571+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:15.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:15.591+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:15.863 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:15.859+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:15.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:15.877 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:15.875+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:15.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:15.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:15.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:15.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:15.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:15.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:15.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:15.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:18.103 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:18.099+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:18.117 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:18.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:18.273 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:18.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:18.276 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:18.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:18.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:18.283+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:18.290 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:18.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:18.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:18.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:19.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:19.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:00:19.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:19.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:19.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:19.379+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:19.382 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:19.379+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:19.393 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:19.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:19.393 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:19.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:19.393 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:19.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:19.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:19.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:19.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:19.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:19.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:19.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:19.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:19.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:19.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:19.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:19.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:19.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:19.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:19.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:20.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:20.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:20.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:20.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:20.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:20.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:20.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:20.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:20.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:20.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:20.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:20.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:20.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:23.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:23.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:23.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:23.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:23.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:23.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:23.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:23.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:23.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:23.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:23.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:23.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:24.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:24.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:24.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:24.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:24.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:24.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:24.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:24.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:24.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:24.867+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:24.874 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:24.871+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:24.900 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:00:24.900 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:00:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:00:24.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:24.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:24.928 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:24.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:24.928 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:24.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:24.928 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:24.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:25.089 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:00:25.087+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:00:25.090 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:00:25.096 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:00:26.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:26.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:26.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:26.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:26.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:26.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:26.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:26.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:26.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:26.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:26.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:26.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:26.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:26.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:26.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:26.699+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:26.712 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:26.711+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:26.731 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:26.727+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:26.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:26.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:26.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:26.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:26.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:26.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:27.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:27.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:27.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:27.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:27.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:27.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:27.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:27.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:27.536 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:27.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:29.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:29.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:29.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:29.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:29.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:29.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:29.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:29.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:29.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:29.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:29.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:29.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:29.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:29.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:29.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:29.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:29.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:29.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:29.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:29.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:29.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:29.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:29.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:29.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:29.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:29.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:30.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:30.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:30.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:30.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:30.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:30.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:30.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:30.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:30.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:30.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:30.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:31.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:00:31.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:00:31.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:31.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:31.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:31.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:31.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:31.749 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:31.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.747+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:31.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:31.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:32.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:32.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:32.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:32.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:32.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:34.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:34.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:34.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:34.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:34.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:34.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:34.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:34.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:34.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:34.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:35.100 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:00:35.100 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:00:35.101 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:00:35.239 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:00:35.235+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:00:35.240 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:00:35.282 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:00:35.766 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:35.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:35.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.791+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:35.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.799+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:35.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.799+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:35.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.799+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:35.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:35.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:35.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:35.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:35.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:35.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:35.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:35.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:35.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:35.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.959+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:00:35.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.959+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:00:35.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.959+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:35.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:35.959+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:37.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:37.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:37.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:37.299+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:37.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:37.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:37.313 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:37.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:37.313 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:37.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:38.870 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:38.867+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:38.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:38.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:39.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:39.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:39.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:39.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:39.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:39.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:39.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:39.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:39.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:39.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:39.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:39.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:40.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:40.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:40.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:40.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:40.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:40.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:40.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:40.119+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:40.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:40.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:40.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:40.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:43.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:43.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:00:43.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:43.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:43.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:43.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:43.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:43.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:43.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:43.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:43.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:43.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:43.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:43.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:45.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:45.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:45.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:45.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.027+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:45.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:45.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:45.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:45.283 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:00:45.283 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:00:45.283 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:00:45.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:45.401 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:00:45.399+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:00:45.406 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:00:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:00:45.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:45.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:45.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:45.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:45.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:45.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:45.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:45.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:45.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:45.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:45.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:45.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:46.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:46.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:46.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:46.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:46.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:46.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:46.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:46.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:46.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:46.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:46.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:46.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:48.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:48.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:48.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:48.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:48.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:48.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:48.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:48.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:48.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:48.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:49.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:49.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:00:49.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:49.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:00:49.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:49.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:49.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:49.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:49.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:49.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:49.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:49.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:49.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:49.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:50.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:50.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:50.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:50.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:50.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:50.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:50.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:50.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:50.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:50.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:50.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:50.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:50.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:50.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:51.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:51.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:51.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:51.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:51.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:51.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:51.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:51.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:51.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:51.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:51.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:51.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:52.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:52.590+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:00:52.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:52.590+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:52.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:52.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:52.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:52.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:52.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:52.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:52.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:52.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:52.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:52.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:53.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:53.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:53.738 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:53.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:53.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:53.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:54.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:54.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:54.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:54.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:00:54.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:00:54.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:54.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:54.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:54.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:54.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:54.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:54.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:00:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:00:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:00:55.579 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:00:55.574+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:00:55.580 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:00:55.616 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:00:56.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:00:56.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:00:56.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:56.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:56.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:56.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:56.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:56.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:56.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:56.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:56.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:57.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:57.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:57.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:57.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:57.230 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:57.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:57.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:57.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:57.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:57.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:00:57.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:57.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:57.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:57.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.630+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:57.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:57.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:57.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:00:57.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:57.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:57.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:57.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:57.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:58.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:58.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:00:58.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:58.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:00:58.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:58.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:00:58.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:58.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:00:58.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:58.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:00:58.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:58.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:00:58.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:00:58.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:00:59.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:59.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:00:59.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:00:59.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.774+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:00:59.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.774+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:00:59.800 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:00:59.800 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:00:59.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:00:59.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:00.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:00.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:00.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:00.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:00.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:00.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:00.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:00.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:00.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:00.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:00.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:01.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:01.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:01.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:01.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:01.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:01.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:01.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:01.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:02.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:02.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:03.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:03.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:03.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:03.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:03.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:03.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:03.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:03.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:04.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:04.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:04.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:04.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:05.007 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:05.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:05.016 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:05.014+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:05.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:05.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:05.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:05.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:05.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:05.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:05.620 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:01:05.620 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:01:05.620 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:01:05.761 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:01:05.758+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:01:05.761 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:01:05.769 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:01:06.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:06.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:06.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:06.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:06.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:06.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:06.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:06.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:07.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:07.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:07.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:07.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:08.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:08.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:08.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:08.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:08.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:08.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:09.025 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:09.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:09.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:09.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:09.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:09.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:09.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:09.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:09.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:09.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:09.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:09.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:09.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:09.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:09.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:09.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:09.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:09.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:10.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:10.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:10.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:10.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:10.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:10.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:10.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:10.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:10.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:10.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:11.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:01:11.240 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.234+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:01:11.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:11.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:11.288 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:11.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.298+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:11.308 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:11.309 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:11.309 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:11.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:11.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:11.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:11.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:11.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:13.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:13.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:13.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:13.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:13.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:13.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:13.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:13.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:13.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:13.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:14.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:14.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:14.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:14.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:14.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:14.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:14.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:14.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:14.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:15.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:14.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:15.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:15.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:15.572 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:15.570+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:15.770 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:01:15.770 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:01:15.771 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:01:15.915 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:01:15.910+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:01:15.916 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:01:15.949 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:01:15.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:15.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:15.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:15.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:15.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:15.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:15.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:15.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:15.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:15.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:15.984 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:15.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:15.985 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:15.982+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:16.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:16.418+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:16.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:16.418+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:17.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:17.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:17.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:17.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:17.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:17.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:17.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:17.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:17.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:17.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:17.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:17.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:17.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:17.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:18.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:18.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:18.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:18.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:18.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:18.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:20.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:20.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:20.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:20.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:20.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:20.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:20.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:20.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.454+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:20.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:20.509 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:20.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:20.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:20.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:20.570 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.566+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:20.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.574+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:20.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:20.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:20.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:20.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:20.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:20.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:20.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:20.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:20.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:20.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:20.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:20.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:20.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:21.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:21.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:21.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:21.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:21.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:21.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:21.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:21.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:21.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:21.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:21.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:21.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:21.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:21.262+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:24.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:24.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:01:24.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:24.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:24.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:24.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:24.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:24.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:24.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:24.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:24.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:24.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:24.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:24.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:25.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:25.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:25.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:25.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:25.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:25.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:25.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:25.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:25.950 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:01:25.951 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:01:25.951 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:01:25.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:25.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:25.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:25.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:26.076 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:01:26.070+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:01:26.077 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:01:26.108 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:01:26.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:26.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:26.214 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.210+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:26.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.210+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:26.215 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.210+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:26.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.514+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:26.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.514+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:26.537 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:26.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:26.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:26.566 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:26.617 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:01:26.618 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:26.618 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.614+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:26.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:26.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:26.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:26.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:26.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:26.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:26.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:26.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:27.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:27.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:27.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:27.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:27.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:27.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:27.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:27.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:27.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:27.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:28.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:28.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:28.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:28.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:28.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:28.922+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:28.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:28.930+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:28.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:28.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:28.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:28.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:28.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:28.942+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:29.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:29.582+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:29.628 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:29.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:29.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:29.630+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:29.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:29.638+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:29.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:29.638+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:29.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:29.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:29.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:29.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:29.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:29.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:31.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:31.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:31.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:31.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:31.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:31.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:31.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:31.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:31.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:31.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:32.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:32.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:32.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:32.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:32.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:32.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:32.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:32.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:32.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:32.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:32.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:32.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:32.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:32.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:32.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:32.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:32.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:32.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:32.698 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:32.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:32.698 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:32.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:32.698 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:32.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:32.698 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:32.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:33.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:33.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:33.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:33.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:33.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:33.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:33.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:33.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:33.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:33.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:33.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:34.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:34.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:34.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:34.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:34.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:34.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:34.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:34.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:34.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:34.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:34.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:34.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:34.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:34.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:34.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:34.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:34.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:34.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:34.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:35.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:35.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:35.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:35.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:35.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:35.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:35.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:35.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:35.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:35.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:35.706 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.702+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:01:35.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:35.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:35.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:35.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:35.764 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:35.764 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:35.764 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:35.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:36.112 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:01:36.113 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:01:36.113 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:01:36.240 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:01:36.233+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:01:36.240 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:01:36.245 INFO:tasks.workunit.client.0.smithi138.stderr:+ sleep 10 2023-10-14T01:01:36.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:36.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:36.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:36.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:36.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:36.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:36.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:36.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:36.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:36.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:37.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:37.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:37.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:37.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:37.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:37.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:37.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:37.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:39.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:39.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:39.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:39.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:39.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:39.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:39.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:39.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:39.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:39.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:39.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:40.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:40.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:40.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:40.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:40.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:40.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:40.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:40.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:40.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:40.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:40.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:40.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:41.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:41.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:41.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:41.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:41.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:41.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:41.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:41.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:41.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:41.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:41.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:41.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:41.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:41.929+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:41.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:41.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:41.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:41.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:41.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:41.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:41.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:41.961+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:41.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:41.961+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:41.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:41.961+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:42.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:42.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:42.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:42.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:42.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:42.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:42.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:42.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:42.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:42.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:42.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:42.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:42.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:43.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:43.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:43.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:43.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:43.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:43.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:43.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:43.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:43.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:43.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:43.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:43.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:43.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:43.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:44.495 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:44.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:44.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:44.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:44.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:44.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:44.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:44.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:44.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:44.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:44.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.521+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:44.537 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.533+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:44.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:44.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:44.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.545+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:44.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.545+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:44.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.545+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:44.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.897+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:44.913 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:44.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:44.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:44.909+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:45.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:45.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:45.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:45.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:45.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:45.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:45.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:45.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:45.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:45.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:46.247 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i++ )) 2023-10-14T01:01:46.247 INFO:tasks.workunit.client.0.smithi138.stderr:+ (( i <= 24 )) 2023-10-14T01:01:46.253 INFO:tasks.workunit.client.0.smithi138.stderr:+ grep 'every 2m' 2023-10-14T01:01:46.260 INFO:tasks.workunit.client.0.smithi138.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-14T01:01:46.399 INFO:tasks.ceph.mgr.x.smithi138.stderr:2023-10-14T01:01:46.393+0000 7f0822305640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:01:46.400 INFO:tasks.workunit.client.0.smithi138.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:01:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T01:01:46.425 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-14T01:01:46.426 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-14T01:01:46.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:46.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:46.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:46.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:46.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:46.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:46.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:46.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:46.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:46.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:46.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:46.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:46.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:46.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:47.161 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-14T01:01:47.225 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=df6af9e7556644af8d34e472e8766dbf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-14T01:01:47.227 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T01:01:47.274 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-14T01:01:47.284 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T01:01:47.285 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:01:47.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:47.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:47.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:47.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:47.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:47.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:47.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:47.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:47.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:47.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:47.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:47.761+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:47.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:47.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:48.048 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:01:48.049 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:01:48.079 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2100,"stamp":"2023-10-14T01:01:47.466363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27164519256,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1981653,"num_read_kb":7682143,"num_write":1940833,"num_write_kb":7763733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80609,"ondisk_log_size":80609,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19319832,"kb_used_data":15523124,"kb_used_omap":1132,"kb_used_meta":3795539,"kb_avail":263795688,"statfs":{"total":289910292480,"available":270126784512,"internally_reserved":0,"allocated":15895678976,"data_stored":20206010232,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1159327,"internal_metadata":3886632801},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,7,5,14,13,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22220800,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5349,"num_read_kb":21288,"num_write":5450,"num_write_kb":21800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001951"},"pg_stats":[{"pgid":"2.7","version":"133'260035","reported_seq":510933,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.072365+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:01:43.072365+0000","last_peered":"2023-10-14T01:01:43.072365+0000","last_clean":"2023-10-14T01:01:43.072365+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:01:43.072365+0000","last_undegraded":"2023-10-14T01:01:43.072365+0000","last_fullsized":"2023-10-14T01:01:43.072365+0000","mapping_epoch":13,"log_start":"131'250002","ondisk_log_start":"131'250002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:09:29.008824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3434185142,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":252772,"num_read_kb":974331,"num_write":244680,"num_write_kb":978478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'260397","reported_seq":512166,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.046270+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:01:43.046270+0000","last_peered":"2023-10-14T01:01:43.046270+0000","last_clean":"2023-10-14T01:01:43.046270+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:01:43.046270+0000","last_undegraded":"2023-10-14T01:01:43.046270+0000","last_fullsized":"2023-10-14T01:01:43.046270+0000","mapping_epoch":13,"log_start":"131'250309","ondisk_log_start":"131'250309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:26.410596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3428647280,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":255244,"num_read_kb":974281,"num_write":245047,"num_write_kb":979815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"133'235569","reported_seq":468740,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.039245+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:01:43.039245+0000","last_peered":"2023-10-14T01:01:43.039245+0000","last_clean":"2023-10-14T01:01:43.039245+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:01:43.039245+0000","last_undegraded":"2023-10-14T01:01:43.039245+0000","last_fullsized":"2023-10-14T01:01:43.039245+0000","mapping_epoch":13,"log_start":"131'225505","ondisk_log_start":"131'225505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-14T23:58:32.220770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3280490898,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":233621,"num_read_kb":919296,"num_write":234951,"num_write_kb":939774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'229875","reported_seq":460401,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.056768+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:01:43.056768+0000","last_peered":"2023-10-14T01:01:43.056768+0000","last_clean":"2023-10-14T01:01:43.056768+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:01:43.056768+0000","last_undegraded":"2023-10-14T01:01:43.056768+0000","last_fullsized":"2023-10-14T01:01:43.056768+0000","mapping_epoch":13,"log_start":"131'219804","ondisk_log_start":"131'219804","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:08:10.692956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3210174595,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":231011,"num_read_kb":907073,"num_write":229270,"num_write_kb":917002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"133'258933","reported_seq":518824,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:42.619768+0000","last_change":"2023-10-13T23:57:08.671048+0000","last_active":"2023-10-14T01:01:42.619768+0000","last_peered":"2023-10-14T01:01:42.619768+0000","last_clean":"2023-10-14T01:01:42.619768+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:01:42.619768+0000","last_undegraded":"2023-10-14T01:01:42.619768+0000","last_fullsized":"2023-10-14T01:01:42.619768+0000","mapping_epoch":13,"log_start":"131'248907","ondisk_log_start":"131'248907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:39:59.117188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3610403135,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":260752,"num_read_kb":1018110,"num_write":257412,"num_write_kb":1029501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"133'250207","reported_seq":496751,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:46.509090+0000","last_change":"2023-10-13T23:57:09.047836+0000","last_active":"2023-10-14T01:01:46.509090+0000","last_peered":"2023-10-14T01:01:46.509090+0000","last_clean":"2023-10-14T01:01:46.509090+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:01:46.509090+0000","last_undegraded":"2023-10-14T01:01:46.509090+0000","last_fullsized":"2023-10-14T01:01:46.509090+0000","mapping_epoch":13,"log_start":"131'240107","ondisk_log_start":"131'240107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:06.210549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3389165837,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":248363,"num_read_kb":957468,"num_write":243024,"num_write_kb":971882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"133'253864","reported_seq":506657,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:46.530917+0000","last_change":"2023-10-13T23:57:09.048148+0000","last_active":"2023-10-14T01:01:46.530917+0000","last_peered":"2023-10-14T01:01:46.530917+0000","last_clean":"2023-10-14T01:01:46.530917+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:01:46.530917+0000","last_undegraded":"2023-10-14T01:01:46.530917+0000","last_fullsized":"2023-10-14T01:01:46.530917+0000","mapping_epoch":13,"log_start":"131'243809","ondisk_log_start":"131'243809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:31:07.537564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3436196089,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":254934,"num_read_kb":977648,"num_write":245096,"num_write_kb":980110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"133'249254","reported_seq":492964,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.055172+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:01:43.055172+0000","last_peered":"2023-10-14T01:01:43.055172+0000","last_clean":"2023-10-14T01:01:43.055172+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:01:43.055172+0000","last_undegraded":"2023-10-14T01:01:43.055172+0000","last_fullsized":"2023-10-14T01:01:43.055172+0000","mapping_epoch":13,"log_start":"131'239207","ondisk_log_start":"131'239207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:40:40.443808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3374665912,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":244830,"num_read_kb":953827,"num_write":241184,"num_write_kb":964623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"53'125","reported_seq":325,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:00:49.316955+0000","last_change":"2023-10-13T23:56:09.871340+0000","last_active":"2023-10-14T01:00:49.316955+0000","last_peered":"2023-10-14T01:00:49.316955+0000","last_clean":"2023-10-14T01:00:49.316955+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:00:49.316955+0000","last_undegraded":"2023-10-14T01:00:49.316955+0000","last_fullsized":"2023-10-14T01:00:49.316955+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_clean_scrub_stamp":"2023-10-13T23:56:08.695661+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:39:25.194520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27163928888,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":1981527,"num_read_kb":7682034,"num_write":1940664,"num_write_kb":7761185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15893606400,"data_stored":20205085296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80484,"ondisk_log_size":80484,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739156,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3688884,"kb_used_data":2918312,"kb_used_omap":212,"kb_used_meta":770347,"kb_avail":90682956,"statfs":{"total":96636764160,"available":92859346944,"internally_reserved":0,"allocated":2988351488,"data_stored":3797586006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":788835572},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.19500000000000001,"5min":0.17999999999999999,"15min":0.14699999999999999},"max":{"1min":3.4780000000000002,"5min":9.4209999999999994,"15min":10.803000000000001},"last":5.165},{"interface":"front","average":{"1min":1.708,"5min":1.0289999999999999,"15min":0.71599999999999997},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":10.847,"5min":10.847,"15min":10.847},"last":0.42999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":2.2069999999999999,"5min":1.2889999999999999,"15min":0.96099999999999997},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":8.266,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.317},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.2609999999999999,"15min":1.1120000000000001},"min":{"1min":0.23400000000000001,"5min":0.19,"15min":0.19},"max":{"1min":5.3090000000000002,"5min":12.928000000000001,"15min":16.736999999999998},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739145,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9626048,"kb_used_data":7759600,"kb_used_omap":563,"kb_used_meta":1865868,"kb_avail":84745792,"statfs":{"total":96636764160,"available":86779691008,"internally_reserved":0,"allocated":7945830400,"data_stored":10101364976,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":577123,"internal_metadata":1910649245},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,2,10,11,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.077,"15min":0.91700000000000004},"min":{"1min":0.189,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.891,"5min":12.369999999999999,"15min":13.109},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.01,"15min":0.91000000000000003},"min":{"1min":0.22,"5min":0.192,"15min":0.192},"max":{"1min":10.856,"5min":12.407,"15min":13.076000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.3049999999999999,"15min":1.139},"min":{"1min":0.254,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":3.552,"5min":14.522,"15min":14.522},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.2070000000000001,"15min":0.92500000000000004},"min":{"1min":0.24099999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2670000000000003,"5min":14.537000000000001,"15min":14.537000000000001},"last":0.39900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739154,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6004900,"kb_used_data":4845212,"kb_used_omap":355,"kb_used_meta":1159324,"kb_avail":88366940,"statfs":{"total":96636764160,"available":90487746560,"internally_reserved":0,"allocated":4961497088,"data_stored":6307059250,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1187147984},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.121,"15min":1.0209999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":8.5239999999999991,"5min":12.855,"15min":12.855},"last":4.7539999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.1120000000000001,"15min":1.204},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.192},"max":{"1min":5.1269999999999998,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.222}]},{"osd":2,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.381,"15min":1.198},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.5380000000000003,"5min":9.1999999999999993,"15min":19.809999999999999},"last":5.1589999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.66000000000000003,"15min":0.81299999999999994},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":2.7229999999999999,"5min":6.3040000000000003,"15min":15.451000000000001},"last":0.34999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4960423936,"data_stored":6306190970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7945838592,"data_stored":10101578040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2987343872,"data_stored":3797316286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:01:48.081 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:01:48.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:48.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:01:48.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:48.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:48.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:48.465+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:48.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:48.477+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:48.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:48.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:48.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:48.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:48.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:48.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:48.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:48.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:48.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:48.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:48.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:48.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:48.809 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:01:48.810 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:01:48.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:48.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:48.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:48.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:48.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:48.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:48.891 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2100,"stamp":"2023-10-14T01:01:47.466363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27164519256,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1981653,"num_read_kb":7682143,"num_write":1940833,"num_write_kb":7763733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80609,"ondisk_log_size":80609,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19319832,"kb_used_data":15523124,"kb_used_omap":1132,"kb_used_meta":3795539,"kb_avail":263795688,"statfs":{"total":289910292480,"available":270126784512,"internally_reserved":0,"allocated":15895678976,"data_stored":20206010232,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1159327,"internal_metadata":3886632801},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,7,5,14,13,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22220800,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5349,"num_read_kb":21288,"num_write":5450,"num_write_kb":21800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001951"},"pg_stats":[{"pgid":"2.7","version":"133'260035","reported_seq":510933,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.072365+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:01:43.072365+0000","last_peered":"2023-10-14T01:01:43.072365+0000","last_clean":"2023-10-14T01:01:43.072365+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:01:43.072365+0000","last_undegraded":"2023-10-14T01:01:43.072365+0000","last_fullsized":"2023-10-14T01:01:43.072365+0000","mapping_epoch":13,"log_start":"131'250002","ondisk_log_start":"131'250002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:09:29.008824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3434185142,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":252772,"num_read_kb":974331,"num_write":244680,"num_write_kb":978478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'260397","reported_seq":512166,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.046270+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:01:43.046270+0000","last_peered":"2023-10-14T01:01:43.046270+0000","last_clean":"2023-10-14T01:01:43.046270+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:01:43.046270+0000","last_undegraded":"2023-10-14T01:01:43.046270+0000","last_fullsized":"2023-10-14T01:01:43.046270+0000","mapping_epoch":13,"log_start":"131'250309","ondisk_log_start":"131'250309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:26.410596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3428647280,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":255244,"num_read_kb":974281,"num_write":245047,"num_write_kb":979815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"133'235569","reported_seq":468740,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.039245+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:01:43.039245+0000","last_peered":"2023-10-14T01:01:43.039245+0000","last_clean":"2023-10-14T01:01:43.039245+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:01:43.039245+0000","last_undegraded":"2023-10-14T01:01:43.039245+0000","last_fullsized":"2023-10-14T01:01:43.039245+0000","mapping_epoch":13,"log_start":"131'225505","ondisk_log_start":"131'225505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-14T23:58:32.220770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3280490898,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":233621,"num_read_kb":919296,"num_write":234951,"num_write_kb":939774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'229875","reported_seq":460401,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.056768+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:01:43.056768+0000","last_peered":"2023-10-14T01:01:43.056768+0000","last_clean":"2023-10-14T01:01:43.056768+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:01:43.056768+0000","last_undegraded":"2023-10-14T01:01:43.056768+0000","last_fullsized":"2023-10-14T01:01:43.056768+0000","mapping_epoch":13,"log_start":"131'219804","ondisk_log_start":"131'219804","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:08:10.692956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3210174595,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":231011,"num_read_kb":907073,"num_write":229270,"num_write_kb":917002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"133'258933","reported_seq":518824,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:42.619768+0000","last_change":"2023-10-13T23:57:08.671048+0000","last_active":"2023-10-14T01:01:42.619768+0000","last_peered":"2023-10-14T01:01:42.619768+0000","last_clean":"2023-10-14T01:01:42.619768+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:01:42.619768+0000","last_undegraded":"2023-10-14T01:01:42.619768+0000","last_fullsized":"2023-10-14T01:01:42.619768+0000","mapping_epoch":13,"log_start":"131'248907","ondisk_log_start":"131'248907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:39:59.117188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3610403135,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":260752,"num_read_kb":1018110,"num_write":257412,"num_write_kb":1029501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"133'250207","reported_seq":496751,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:46.509090+0000","last_change":"2023-10-13T23:57:09.047836+0000","last_active":"2023-10-14T01:01:46.509090+0000","last_peered":"2023-10-14T01:01:46.509090+0000","last_clean":"2023-10-14T01:01:46.509090+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:01:46.509090+0000","last_undegraded":"2023-10-14T01:01:46.509090+0000","last_fullsized":"2023-10-14T01:01:46.509090+0000","mapping_epoch":13,"log_start":"131'240107","ondisk_log_start":"131'240107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:06.210549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3389165837,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":248363,"num_read_kb":957468,"num_write":243024,"num_write_kb":971882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"133'253864","reported_seq":506657,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:46.530917+0000","last_change":"2023-10-13T23:57:09.048148+0000","last_active":"2023-10-14T01:01:46.530917+0000","last_peered":"2023-10-14T01:01:46.530917+0000","last_clean":"2023-10-14T01:01:46.530917+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:01:46.530917+0000","last_undegraded":"2023-10-14T01:01:46.530917+0000","last_fullsized":"2023-10-14T01:01:46.530917+0000","mapping_epoch":13,"log_start":"131'243809","ondisk_log_start":"131'243809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:31:07.537564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3436196089,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":254934,"num_read_kb":977648,"num_write":245096,"num_write_kb":980110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"133'249254","reported_seq":492964,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:43.055172+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:01:43.055172+0000","last_peered":"2023-10-14T01:01:43.055172+0000","last_clean":"2023-10-14T01:01:43.055172+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:01:43.055172+0000","last_undegraded":"2023-10-14T01:01:43.055172+0000","last_fullsized":"2023-10-14T01:01:43.055172+0000","mapping_epoch":13,"log_start":"131'239207","ondisk_log_start":"131'239207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:40:40.443808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3374665912,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":244830,"num_read_kb":953827,"num_write":241184,"num_write_kb":964623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"53'125","reported_seq":325,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:00:49.316955+0000","last_change":"2023-10-13T23:56:09.871340+0000","last_active":"2023-10-14T01:00:49.316955+0000","last_peered":"2023-10-14T01:00:49.316955+0000","last_clean":"2023-10-14T01:00:49.316955+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:00:49.316955+0000","last_undegraded":"2023-10-14T01:00:49.316955+0000","last_fullsized":"2023-10-14T01:00:49.316955+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_clean_scrub_stamp":"2023-10-13T23:56:08.695661+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:39:25.194520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27163928888,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":1981527,"num_read_kb":7682034,"num_write":1940664,"num_write_kb":7761185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15893606400,"data_stored":20205085296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80484,"ondisk_log_size":80484,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739156,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3688884,"kb_used_data":2918312,"kb_used_omap":212,"kb_used_meta":770347,"kb_avail":90682956,"statfs":{"total":96636764160,"available":92859346944,"internally_reserved":0,"allocated":2988351488,"data_stored":3797586006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":788835572},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.19500000000000001,"5min":0.17999999999999999,"15min":0.14699999999999999},"max":{"1min":3.4780000000000002,"5min":9.4209999999999994,"15min":10.803000000000001},"last":5.165},{"interface":"front","average":{"1min":1.708,"5min":1.0289999999999999,"15min":0.71599999999999997},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":10.847,"5min":10.847,"15min":10.847},"last":0.42999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":2.2069999999999999,"5min":1.2889999999999999,"15min":0.96099999999999997},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":8.266,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.317},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.2609999999999999,"15min":1.1120000000000001},"min":{"1min":0.23400000000000001,"5min":0.19,"15min":0.19},"max":{"1min":5.3090000000000002,"5min":12.928000000000001,"15min":16.736999999999998},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739145,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9626048,"kb_used_data":7759600,"kb_used_omap":563,"kb_used_meta":1865868,"kb_avail":84745792,"statfs":{"total":96636764160,"available":86779691008,"internally_reserved":0,"allocated":7945830400,"data_stored":10101364976,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":577123,"internal_metadata":1910649245},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,2,10,11,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.077,"15min":0.91700000000000004},"min":{"1min":0.189,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.891,"5min":12.369999999999999,"15min":13.109},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.01,"15min":0.91000000000000003},"min":{"1min":0.22,"5min":0.192,"15min":0.192},"max":{"1min":10.856,"5min":12.407,"15min":13.076000000000001},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.3049999999999999,"15min":1.139},"min":{"1min":0.254,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":3.552,"5min":14.522,"15min":14.522},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.2070000000000001,"15min":0.92500000000000004},"min":{"1min":0.24099999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2670000000000003,"5min":14.537000000000001,"15min":14.537000000000001},"last":0.39900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739154,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6004900,"kb_used_data":4845212,"kb_used_omap":355,"kb_used_meta":1159324,"kb_avail":88366940,"statfs":{"total":96636764160,"available":90487746560,"internally_reserved":0,"allocated":4961497088,"data_stored":6307059250,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1187147984},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.121,"15min":1.0209999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":8.5239999999999991,"5min":12.855,"15min":12.855},"last":4.7539999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.1120000000000001,"15min":1.204},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.192},"max":{"1min":5.1269999999999998,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.222}]},{"osd":2,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.381,"15min":1.198},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.5380000000000003,"5min":9.1999999999999993,"15min":19.809999999999999},"last":5.1589999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.66000000000000003,"15min":0.81299999999999994},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":2.7229999999999999,"5min":6.3040000000000003,"15min":15.451000000000001},"last":0.34999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4960423936,"data_stored":6306190970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7945838592,"data_stored":10101578040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2987343872,"data_stored":3797316286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:01:48.892 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T01:01:48.893 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:01:49.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:49.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:49.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:49.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:49.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:49.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:49.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:49.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:49.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:49.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:49.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:49.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:49.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:49.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:49.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:49.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:49.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:49.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:49.917 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:01:49.917 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:01:49.999 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2101,"stamp":"2023-10-14T01:01:49.466740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27171760984,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1983394,"num_read_kb":7689079,"num_write":1942613,"num_write_kb":7770853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80566,"ondisk_log_size":80566,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19383872,"kb_used_data":15538596,"kb_used_omap":1132,"kb_used_meta":3844115,"kb_avail":263731648,"statfs":{"total":289910292480,"available":270061207552,"internally_reserved":0,"allocated":15911522304,"data_stored":20221853560,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1159327,"internal_metadata":3936374625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,11,12,25,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27049984,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6475,"num_read_kb":25772,"num_write":6640,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002003"},"pg_stats":[{"pgid":"2.7","version":"133'260386","reported_seq":511596,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:48.142314+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:01:48.142314+0000","last_peered":"2023-10-14T01:01:48.142314+0000","last_clean":"2023-10-14T01:01:48.142314+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:01:48.142314+0000","last_undegraded":"2023-10-14T01:01:48.142314+0000","last_fullsized":"2023-10-14T01:01:48.142314+0000","mapping_epoch":13,"log_start":"131'250302","ondisk_log_start":"131'250302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:09:29.008824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3435471286,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":253079,"num_read_kb":975551,"num_write":244999,"num_write_kb":979754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'260738","reported_seq":512803,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:48.136133+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:01:48.136133+0000","last_peered":"2023-10-14T01:01:48.136133+0000","last_clean":"2023-10-14T01:01:48.136133+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:01:48.136133+0000","last_undegraded":"2023-10-14T01:01:48.136133+0000","last_fullsized":"2023-10-14T01:01:48.136133+0000","mapping_epoch":13,"log_start":"131'250709","ondisk_log_start":"131'250709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:26.410596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3429921136,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":255537,"num_read_kb":975445,"num_write":245359,"num_write_kb":981063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"133'235831","reported_seq":469287,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:48.126145+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:01:48.126145+0000","last_peered":"2023-10-14T01:01:48.126145+0000","last_clean":"2023-10-14T01:01:48.126145+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:01:48.126145+0000","last_undegraded":"2023-10-14T01:01:48.126145+0000","last_fullsized":"2023-10-14T01:01:48.126145+0000","mapping_epoch":13,"log_start":"131'225805","ondisk_log_start":"131'225805","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-14T23:58:32.220770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3281555858,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":233905,"num_read_kb":920432,"num_write":235213,"num_write_kb":940822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'230163","reported_seq":460956,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:48.126571+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:01:48.126571+0000","last_peered":"2023-10-14T01:01:48.126571+0000","last_clean":"2023-10-14T01:01:48.126571+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:01:48.126571+0000","last_undegraded":"2023-10-14T01:01:48.126571+0000","last_fullsized":"2023-10-14T01:01:48.126571+0000","mapping_epoch":13,"log_start":"131'220104","ondisk_log_start":"131'220104","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:08:10.692956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3211341955,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":231278,"num_read_kb":908137,"num_write":229558,"num_write_kb":918154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"133'259262","reported_seq":519456,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:47.667577+0000","last_change":"2023-10-13T23:57:08.671048+0000","last_active":"2023-10-14T01:01:47.667577+0000","last_peered":"2023-10-14T01:01:47.667577+0000","last_clean":"2023-10-14T01:01:47.667577+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:01:47.667577+0000","last_undegraded":"2023-10-14T01:01:47.667577+0000","last_fullsized":"2023-10-14T01:01:47.667577+0000","mapping_epoch":13,"log_start":"131'249207","ondisk_log_start":"131'249207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:39:59.117188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3611742527,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":261052,"num_read_kb":1019306,"num_write":257741,"num_write_kb":1030817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"133'250207","reported_seq":496751,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:46.509090+0000","last_change":"2023-10-13T23:57:09.047836+0000","last_active":"2023-10-14T01:01:46.509090+0000","last_peered":"2023-10-14T01:01:46.509090+0000","last_clean":"2023-10-14T01:01:46.509090+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:01:46.509090+0000","last_undegraded":"2023-10-14T01:01:46.509090+0000","last_fullsized":"2023-10-14T01:01:46.509090+0000","mapping_epoch":13,"log_start":"131'240107","ondisk_log_start":"131'240107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:06.210549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3389165837,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":248363,"num_read_kb":957468,"num_write":243024,"num_write_kb":971882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"133'253864","reported_seq":506657,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:46.530917+0000","last_change":"2023-10-13T23:57:09.048148+0000","last_active":"2023-10-14T01:01:46.530917+0000","last_peered":"2023-10-14T01:01:46.530917+0000","last_clean":"2023-10-14T01:01:46.530917+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:01:46.530917+0000","last_undegraded":"2023-10-14T01:01:46.530917+0000","last_fullsized":"2023-10-14T01:01:46.530917+0000","mapping_epoch":13,"log_start":"131'243809","ondisk_log_start":"131'243809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:31:07.537564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3436196089,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":254934,"num_read_kb":977648,"num_write":245096,"num_write_kb":980110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"133'249540","reported_seq":493540,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:48.152000+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:01:48.152000+0000","last_peered":"2023-10-14T01:01:48.152000+0000","last_clean":"2023-10-14T01:01:48.152000+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:01:48.152000+0000","last_undegraded":"2023-10-14T01:01:48.152000+0000","last_fullsized":"2023-10-14T01:01:48.152000+0000","mapping_epoch":13,"log_start":"131'239507","ondisk_log_start":"131'239507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:40:40.443808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3375775928,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":245120,"num_read_kb":954983,"num_write":241454,"num_write_kb":965703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"53'125","reported_seq":325,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:00:49.316955+0000","last_change":"2023-10-13T23:56:09.871340+0000","last_active":"2023-10-14T01:00:49.316955+0000","last_peered":"2023-10-14T01:00:49.316955+0000","last_clean":"2023-10-14T01:00:49.316955+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:00:49.316955+0000","last_undegraded":"2023-10-14T01:00:49.316955+0000","last_fullsized":"2023-10-14T01:00:49.316955+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_clean_scrub_stamp":"2023-10-13T23:56:08.695661+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:39:25.194520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27171170616,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":1983268,"num_read_kb":7688970,"num_write":1942444,"num_write_kb":7768305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15909556224,"data_stored":20221035120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80441,"ondisk_log_size":80441,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739156,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3688884,"kb_used_data":2918312,"kb_used_omap":212,"kb_used_meta":770347,"kb_avail":90682956,"statfs":{"total":96636764160,"available":92859346944,"internally_reserved":0,"allocated":2988351488,"data_stored":3797586006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":788835572},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.19500000000000001,"5min":0.17999999999999999,"15min":0.14699999999999999},"max":{"1min":3.4780000000000002,"5min":9.4209999999999994,"15min":10.803000000000001},"last":5.165},{"interface":"front","average":{"1min":1.708,"5min":1.0289999999999999,"15min":0.71599999999999997},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":10.847,"5min":10.847,"15min":10.847},"last":0.42999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":2.2069999999999999,"5min":1.2889999999999999,"15min":0.96099999999999997},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":8.266,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.317},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.2609999999999999,"15min":1.1120000000000001},"min":{"1min":0.23400000000000001,"5min":0.19,"15min":0.19},"max":{"1min":5.3090000000000002,"5min":12.928000000000001,"15min":16.736999999999998},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739146,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9661704,"kb_used_data":7768896,"kb_used_omap":563,"kb_used_meta":1892236,"kb_avail":84710136,"statfs":{"total":96636764160,"available":86743179264,"internally_reserved":0,"allocated":7955349504,"data_stored":10110884080,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":577123,"internal_metadata":1937650077},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,9,17,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.077,"15min":0.91700000000000004},"min":{"1min":0.189,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.891,"5min":12.369999999999999,"15min":13.109},"last":0.22},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.01,"15min":0.91000000000000003},"min":{"1min":0.22,"5min":0.192,"15min":0.192},"max":{"1min":10.856,"5min":12.407,"15min":13.076000000000001},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.3049999999999999,"15min":1.139},"min":{"1min":0.254,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":3.552,"5min":14.522,"15min":14.522},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.2070000000000001,"15min":0.92500000000000004},"min":{"1min":0.24099999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2670000000000003,"5min":14.537000000000001,"15min":14.537000000000001},"last":0.86499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739155,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6033284,"kb_used_data":4851388,"kb_used_omap":355,"kb_used_meta":1181532,"kb_avail":88338556,"statfs":{"total":96636764160,"available":90458681344,"internally_reserved":0,"allocated":4967821312,"data_stored":6313383474,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1209888976},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.121,"15min":1.0209999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":8.5239999999999991,"5min":12.855,"15min":12.855},"last":0.219},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.1120000000000001,"15min":1.204},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.192},"max":{"1min":5.1269999999999998,"5min":16.655000000000001,"15min":16.655000000000001},"last":0.23799999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.381,"15min":1.198},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.5380000000000003,"5min":9.1999999999999993,"15min":19.809999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.66000000000000003,"15min":0.81299999999999994},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":2.7229999999999999,"5min":6.3040000000000003,"15min":15.451000000000001},"last":2.9340000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4966612992,"data_stored":6312380026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7955599360,"data_stored":10111338808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2987343872,"data_stored":3797316286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:01:50.000 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T01:01:50.764 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:01:50.765 INFO:teuthology.orchestra.run.smithi138.stdout:{"epoch":133,"fsid":"be73fd16-7c0f-4418-ab81-68a6185f41a8","created":"2023-10-13T23:55:57.637104+0000","modified":"2023-10-14T01:00:49.288278+0000","last_up_change":"2023-10-13T23:56:06.677931+0000","last_in_change":"2023-10-13T23:55:58.541401+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-13T23:56:08.413739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-13T23:56:12.839467+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":43,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5f50877-3e63-4f7c-a58b-da90780b0e61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6805","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6807","nonce":4261051765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6809","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6811","nonce":4261051765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6814","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6815","nonce":4261051765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6812","nonce":4261051765},{"type":"v1","addr":"172.21.15.138:6813","nonce":4261051765}]},"public_addr":"172.21.15.138:6807/4261051765","cluster_addr":"172.21.15.138:6811/4261051765","heartbeat_back_addr":"172.21.15.138:6815/4261051765","heartbeat_front_addr":"172.21.15.138:6813/4261051765","state":["exists","up"]},{"osd":1,"uuid":"6978755c-6c5b-4d64-a2de-bac99a43601c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6816","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6817","nonce":4204819583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6818","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6819","nonce":4204819583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6822","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6823","nonce":4204819583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6820","nonce":4204819583},{"type":"v1","addr":"172.21.15.138:6821","nonce":4204819583}]},"public_addr":"172.21.15.138:6817/4204819583","cluster_addr":"172.21.15.138:6819/4204819583","heartbeat_back_addr":"172.21.15.138:6823/4204819583","heartbeat_front_addr":"172.21.15.138:6821/4204819583","state":["exists","up"]},{"osd":2,"uuid":"c105e221-238a-4079-8e50-7ac981476525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6800","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6801","nonce":2835165467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6802","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6803","nonce":2835165467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6808","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6810","nonce":2835165467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.138:6804","nonce":2835165467},{"type":"v1","addr":"172.21.15.138:6806","nonce":2835165467}]},"public_addr":"172.21.15.138:6801/2835165467","cluster_addr":"172.21.15.138:6803/2835165467","heartbeat_back_addr":"172.21.15.138:6810/2835165467","heartbeat_front_addr":"172.21.15.138:6806/2835165467","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.800709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:03.799864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-13T23:56:04.109910+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.138:0/393253116":"2023-10-14T01:42:36.873860+0000","172.21.15.138:0/263115127":"2023-10-14T01:41:22.429231+0000","172.21.15.138:0/2551040678":"2023-10-14T01:40:10.703602+0000","172.21.15.138:0/3912908247":"2023-10-14T01:38:58.493549+0000","172.21.15.138:0/2550805914":"2023-10-14T01:36:34.036444+0000","172.21.15.138:0/801064916":"2023-10-14T01:35:06.098807+0000","172.21.15.138:0/2938574976":"2023-10-14T01:30:05.073766+0000","172.21.15.138:0/3862910917":"2023-10-14T01:28:53.051872+0000","172.21.15.138:0/4079301895":"2023-10-14T01:42:22.421743+0000","172.21.15.138:0/2983772729":"2023-10-14T01:27:41.199944+0000","172.21.15.138:0/3331146725":"2023-10-14T01:08:01.661534+0000","172.21.15.138:0/1006714013":"2023-10-14T01:05:37.004191+0000","172.21.15.138:0/2731291358":"2023-10-14T01:22:37.926004+0000","172.21.15.138:0/1862252240":"2023-10-14T01:06:49.841200+0000","172.21.15.138:0/296162553":"2023-10-14T01:36:18.648931+0000","172.21.15.138:0/2993341611":"2023-10-14T01:03:10.841782+0000","172.21.15.138:0/2845967498":"2023-10-14T01:01:58.173825+0000","172.21.15.138:0/3463294261":"2023-10-14T01:20:24.917839+0000","172.21.15.138:0/1128175262":"2023-10-14T01:32:41.709006+0000","172.21.15.138:0/2888421112":"2023-10-14T01:09:29.553503+0000","172.21.15.138:0/967096462":"2023-10-14T01:16:40.399970+0000","172.21.15.138:0/359867429":"2023-10-14T01:09:14.145467+0000","172.21.15.138:0/1611482965":"2023-10-14T01:26:29.248963+0000","172.21.15.138:0/2052584210":"2023-10-14T01:14:16.363538+0000","172.21.15.138:0/947637180":"2023-10-14T01:11:54.452830+0000","172.21.15.138:0/92831769":"2023-10-14T01:04:23.914772+0000","172.21.15.138:0/1188183334":"2023-10-14T01:13:05.857194+0000","172.21.15.138:0/3062435210":"2023-10-14T01:15:28.837178+0000","172.21.15.138:0/762069660":"2023-10-14T01:17:42.210750+0000","172.21.15.138:0/3271336887":"2023-10-14T01:37:46.035897+0000","172.21.15.138:0/3135251554":"2023-10-14T01:17:56.711895+0000","172.21.15.138:0/2872413020":"2023-10-14T01:33:53.952991+0000","172.21.15.138:0/112034370":"2023-10-14T01:31:29.225782+0000","172.21.15.138:0/3179505995":"2023-10-14T01:18:57.069622+0000","172.21.15.138:0/3864440564":"2023-10-14T01:10:41.623887+0000","172.21.15.138:0/327634990":"2023-10-14T01:20:09.308645+0000","172.21.15.138:0/3820507230":"2023-10-14T01:21:25.870329+0000","172.21.15.138:0/2561721673":"2023-10-14T01:25:01.850739+0000","172.21.15.138:0/2336971439":"2023-10-14T01:23:49.567838+0000","172.21.15.138:0/1887320117":"2023-10-14T01:25:17.189501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T01:01:51.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:51.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:51.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:51.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:51.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:51.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:51.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:51.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:51.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:51.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:51.830 INFO:tasks.ceph:Scrubbing osd.0 2023-10-14T01:01:51.831 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:01:51.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:51.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:51.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:51.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:52.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:52.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:52.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:52.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:52.293 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2023-10-14T01:01:52.294 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:01:52.294 INFO:teuthology.orchestra.run.smithi138.stdout:} 2023-10-14T01:01:52.330 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-14T01:01:53.208 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 0 to deep-scrub 2023-10-14T01:01:53.278 INFO:tasks.ceph:Scrubbing osd.1 2023-10-14T01:01:53.279 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:01:53.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:01:53.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:53.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:53.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:53.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:53.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:53.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:53.899 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2023-10-14T01:01:53.899 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:01:53.899 INFO:teuthology.orchestra.run.smithi138.stdout:} 2023-10-14T01:01:53.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:53.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:01:53.963 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-14T01:01:54.813 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 1 to deep-scrub 2023-10-14T01:01:54.860 INFO:tasks.ceph:Scrubbing osd.2 2023-10-14T01:01:54.861 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:01:55.068 INFO:teuthology.orchestra.run.smithi138.stdout:{ 2023-10-14T01:01:55.069 INFO:teuthology.orchestra.run.smithi138.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:01:55.069 INFO:teuthology.orchestra.run.smithi138.stdout:} 2023-10-14T01:01:55.098 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-14T01:01:55.377 INFO:teuthology.orchestra.run.smithi138.stderr:instructed osd(s) 2 to deep-scrub 2023-10-14T01:01:55.422 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:01:56.012 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:01:56.013 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:01:56.043 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2104,"stamp":"2023-10-14T01:01:55.470287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27181083480,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1985591,"num_read_kb":7697823,"num_write":1944887,"num_write_kb":7779949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80499,"ondisk_log_size":80499,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19443248,"kb_used_data":15556568,"kb_used_omap":1132,"kb_used_meta":3885523,"kb_avail":263672272,"statfs":{"total":289910292480,"available":270000406528,"internally_reserved":0,"allocated":15929925632,"data_stored":20240256888,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1159327,"internal_metadata":3978776417},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,14,13,20,6,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18935808,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4501,"num_read_kb":17912,"num_write":4634,"num_write_kb":18536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004573"},"pg_stats":[{"pgid":"2.7","version":"133'260699","reported_seq":512208,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:53.247024+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:01:53.247024+0000","last_peered":"2023-10-14T01:01:53.247024+0000","last_clean":"2023-10-14T01:01:53.247024+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:01:53.247024+0000","last_undegraded":"2023-10-14T01:01:53.247024+0000","last_fullsized":"2023-10-14T01:01:53.247024+0000","mapping_epoch":13,"log_start":"131'250602","ondisk_log_start":"131'250602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:09:29.008824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3436675510,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":253374,"num_read_kb":976727,"num_write":245296,"num_write_kb":980942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'261053","reported_seq":513423,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:53.245802+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:01:53.245802+0000","last_peered":"2023-10-14T01:01:53.245802+0000","last_clean":"2023-10-14T01:01:53.245802+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:01:53.245802+0000","last_undegraded":"2023-10-14T01:01:53.245802+0000","last_fullsized":"2023-10-14T01:01:53.245802+0000","mapping_epoch":13,"log_start":"131'251009","ondisk_log_start":"131'251009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:26.410596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3431145840,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":255835,"num_read_kb":976629,"num_write":245658,"num_write_kb":982259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"133'236102","reported_seq":469821,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:53.253036+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:01:53.253036+0000","last_peered":"2023-10-14T01:01:53.253036+0000","last_clean":"2023-10-14T01:01:53.253036+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:01:53.253036+0000","last_undegraded":"2023-10-14T01:01:53.253036+0000","last_fullsized":"2023-10-14T01:01:53.253036+0000","mapping_epoch":13,"log_start":"131'226005","ondisk_log_start":"131'226005","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-14T23:58:32.220770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3282665874,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":234166,"num_read_kb":921464,"num_write":235484,"num_write_kb":941906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"133'230423","reported_seq":461459,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:53.214461+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:01:53.214461+0000","last_peered":"2023-10-14T01:01:53.214461+0000","last_clean":"2023-10-14T01:01:53.214461+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:01:53.214461+0000","last_undegraded":"2023-10-14T01:01:53.214461+0000","last_fullsized":"2023-10-14T01:01:53.214461+0000","mapping_epoch":13,"log_start":"131'220404","ondisk_log_start":"131'220404","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:08:10.692956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3212398723,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":231521,"num_read_kb":909109,"num_write":229818,"num_write_kb":919194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"133'259568","reported_seq":520050,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:52.719397+0000","last_change":"2023-10-13T23:57:08.671048+0000","last_active":"2023-10-14T01:01:52.719397+0000","last_peered":"2023-10-14T01:01:52.719397+0000","last_clean":"2023-10-14T01:01:52.719397+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:01:52.719397+0000","last_undegraded":"2023-10-14T01:01:52.719397+0000","last_fullsized":"2023-10-14T01:01:52.719397+0000","mapping_epoch":13,"log_start":"131'249507","ondisk_log_start":"131'249507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:39:59.117188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3612983615,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":261346,"num_read_kb":1020478,"num_write":258046,"num_write_kb":1032037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"133'250514","reported_seq":497328,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:51.540929+0000","last_change":"2023-10-13T23:57:09.047836+0000","last_active":"2023-10-14T01:01:51.540929+0000","last_peered":"2023-10-14T01:01:51.540929+0000","last_clean":"2023-10-14T01:01:51.540929+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:01:51.540929+0000","last_undegraded":"2023-10-14T01:01:51.540929+0000","last_fullsized":"2023-10-14T01:01:51.540929+0000","mapping_epoch":13,"log_start":"131'240507","ondisk_log_start":"131'240507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:06.210549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3390390541,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":248633,"num_read_kb":958544,"num_write":243323,"num_write_kb":973078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"133'254139","reported_seq":507196,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:51.565263+0000","last_change":"2023-10-13T23:57:09.048148+0000","last_active":"2023-10-14T01:01:51.565263+0000","last_peered":"2023-10-14T01:01:51.565263+0000","last_clean":"2023-10-14T01:01:51.565263+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:01:51.565263+0000","last_undegraded":"2023-10-14T01:01:51.565263+0000","last_fullsized":"2023-10-14T01:01:51.565263+0000","mapping_epoch":13,"log_start":"131'244109","ondisk_log_start":"131'244109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:31:07.537564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3437302009,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":255192,"num_read_kb":978668,"num_write":245361,"num_write_kb":981170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"133'249826","reported_seq":494105,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:53.184850+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:01:53.184850+0000","last_peered":"2023-10-14T01:01:53.184850+0000","last_clean":"2023-10-14T01:01:53.184850+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:01:53.184850+0000","last_undegraded":"2023-10-14T01:01:53.184850+0000","last_fullsized":"2023-10-14T01:01:53.184850+0000","mapping_epoch":13,"log_start":"131'239807","ondisk_log_start":"131'239807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:40:40.443808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3376931000,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":245398,"num_read_kb":956095,"num_write":241732,"num_write_kb":966815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"53'125","reported_seq":325,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:00:49.316955+0000","last_change":"2023-10-13T23:56:09.871340+0000","last_active":"2023-10-14T01:00:49.316955+0000","last_peered":"2023-10-14T01:00:49.316955+0000","last_clean":"2023-10-14T01:00:49.316955+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:00:49.316955+0000","last_undegraded":"2023-10-14T01:00:49.316955+0000","last_fullsized":"2023-10-14T01:00:49.316955+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:08.695661+0000","last_clean_scrub_stamp":"2023-10-13T23:56:08.695661+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:39:25.194520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27180493112,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":1985465,"num_read_kb":7697714,"num_write":1944718,"num_write_kb":7777401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15928205312,"data_stored":20239684208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80374,"ondisk_log_size":80374,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739157,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3679728,"kb_used_data":2921640,"kb_used_omap":212,"kb_used_meta":757867,"kb_avail":90692112,"statfs":{"total":96636764160,"available":92868722688,"internally_reserved":0,"allocated":2991759360,"data_stored":3800993878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":776056052},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,6,5,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.19500000000000001,"5min":0.17999999999999999,"15min":0.14699999999999999},"max":{"1min":3.4780000000000002,"5min":9.4209999999999994,"15min":10.803000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.708,"5min":1.0289999999999999,"15min":0.71599999999999997},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":10.847,"5min":10.847,"15min":10.847},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:00:55 2023","interfaces":[{"interface":"back","average":{"1min":2.2069999999999999,"5min":1.2889999999999999,"15min":0.96099999999999997},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":8.266,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.2609999999999999,"15min":1.1120000000000001},"min":{"1min":0.23400000000000001,"5min":0.19,"15min":0.19},"max":{"1min":5.3090000000000002,"5min":12.928000000000001,"15min":16.736999999999998},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739147,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9676288,"kb_used_data":7777912,"kb_used_omap":563,"kb_used_meta":1897804,"kb_avail":84695552,"statfs":{"total":96636764160,"available":86728245248,"internally_reserved":0,"allocated":7964581888,"data_stored":10120116464,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":577123,"internal_metadata":1943351709},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,12,9,13,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:01:08 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.077,"15min":0.91700000000000004},"min":{"1min":0.189,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.891,"5min":12.369999999999999,"15min":13.109},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.01,"15min":0.91000000000000003},"min":{"1min":0.22,"5min":0.192,"15min":0.192},"max":{"1min":10.856,"5min":12.407,"15min":13.076000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.3049999999999999,"15min":1.139},"min":{"1min":0.254,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":3.552,"5min":14.522,"15min":14.522},"last":0.315},{"interface":"front","average":{"1min":1.161,"5min":1.2070000000000001,"15min":0.92500000000000004},"min":{"1min":0.24099999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2670000000000003,"5min":14.537000000000001,"15min":14.537000000000001},"last":0.26200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739156,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6087232,"kb_used_data":4857016,"kb_used_omap":355,"kb_used_meta":1229852,"kb_avail":88284608,"statfs":{"total":96636764160,"available":90403438592,"internally_reserved":0,"allocated":4973584384,"data_stored":6319146546,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1259368656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.121,"15min":1.0209999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":8.5239999999999991,"5min":12.855,"15min":12.855},"last":3.8500000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":1.1120000000000001,"15min":1.204},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.192},"max":{"1min":5.1269999999999998,"5min":16.655000000000001,"15min":16.655000000000001},"last":3.972}]},{"osd":2,"last update":"Sat Oct 14 01:01:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.381,"15min":1.198},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.5380000000000003,"5min":9.1999999999999993,"15min":19.809999999999999},"last":8.4350000000000005},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.66000000000000003,"15min":0.81299999999999994},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":2.7229999999999999,"5min":6.3040000000000003,"15min":15.451000000000001},"last":3.8929999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4972417024,"data_stored":6318184058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7964975104,"data_stored":10120714552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2990813184,"data_stored":3800785598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:01:56.046 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.047 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.047 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.048 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.048 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.049 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.049 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.049 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.050 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-13T23:56:08.695661+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=8, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:01:56.050 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:01:56.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:56.973+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:01:56.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:56.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:01:56.997 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:56.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:01:57.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:57.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:01:57.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:57.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:01:57.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:01:57.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:01:57.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:57.985+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:57.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:57.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:57.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:57.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:59.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:59.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:01:59.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:59.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:01:59.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:59.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:01:59.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:59.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:01:59.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:59.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:01:59.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:59.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:01:59.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:01:59.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:01.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:01.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:01.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:01.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:01.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:01.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:01.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:01.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:02.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:02.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:02.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:02.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:02.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:02.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:02.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:02.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:02.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:02.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:03.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:03.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:03.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:03.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:03.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:03.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:03.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:03.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:04.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:04.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:04.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:04.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:04.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:04.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:04.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:04.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:04.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:04.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:05.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:05.141+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:05.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:05.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:05.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:05.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:02:05.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:05.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:06.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:06.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:06.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:06.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:06.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:06.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:06.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:06.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:06.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:06.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:07.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:07.021+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:07.027 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:07.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:07.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:07.037+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:07.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:07.037+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:02:07.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:07.037+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:08.219 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:08.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:08.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:08.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:08.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:08.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:08.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:08.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:08.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:08.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:10.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:10.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:10.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:10.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:10.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:10.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:02:10.109 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:10.105+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:12.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:12.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:12.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:12.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:12.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:12.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:12.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:12.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:12.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:12.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:12.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:12.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:12.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:12.869+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:13.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:13.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:13.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:13.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:13.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:13.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:02:13.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:13.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.733+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:13.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:13.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:13.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:13.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:13.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:13.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:13.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:13.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:13.769+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:02:13.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:13.769+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:02:13.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:13.773+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:13.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:13.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:13.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:13.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:13.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:13.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:02:13.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:13.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:16.052 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:02:16.908 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:02:16.908 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:02:16.988 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2115,"stamp":"2023-10-14T01:02:15.473868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27213179736,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":1993300,"num_read_kb":7728503,"num_write":1952793,"num_write_kb":7811573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80503,"ondisk_log_size":80503,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19763464,"kb_used_data":15620200,"kb_used_omap":1132,"kb_used_meta":4142099,"kb_avail":263352056,"statfs":{"total":289910292480,"available":269672505344,"internally_reserved":0,"allocated":15995084800,"data_stored":20305393748,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1159327,"internal_metadata":4241510241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,11,6,13,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21843968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5284,"num_read_kb":21036,"num_write":5384,"num_write_kb":21536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002296"},"pg_stats":[{"pgid":"2.7","version":"134'261756","reported_seq":514213,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:13.961354+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:02:13.961354+0000","last_peered":"2023-10-14T01:02:13.961354+0000","last_clean":"2023-10-14T01:02:13.961354+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:02:13.961354+0000","last_undegraded":"2023-10-14T01:02:13.961354+0000","last_fullsized":"2023-10-14T01:02:13.961354+0000","mapping_epoch":13,"log_start":"131'251702","ondisk_log_start":"131'251702","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3440734646,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":254305,"num_read_kb":980431,"num_write":246295,"num_write_kb":984938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'262124","reported_seq":515495,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:13.961078+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:02:13.961078+0000","last_peered":"2023-10-14T01:02:13.961078+0000","last_clean":"2023-10-14T01:02:13.961078+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:02:13.961078+0000","last_undegraded":"2023-10-14T01:02:13.961078+0000","last_fullsized":"2023-10-14T01:02:13.961078+0000","mapping_epoch":13,"log_start":"131'252109","ondisk_log_start":"131'252109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3435258224,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":256820,"num_read_kb":980545,"num_write":246669,"num_write_kb":986303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'237069","reported_seq":471699,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:13.957219+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:02:13.957219+0000","last_peered":"2023-10-14T01:02:13.957219+0000","last_clean":"2023-10-14T01:02:13.957219+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:02:13.957219+0000","last_undegraded":"2023-10-14T01:02:13.957219+0000","last_fullsized":"2023-10-14T01:02:13.957219+0000","mapping_epoch":13,"log_start":"131'227005","ondisk_log_start":"131'227005","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3286585746,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":235080,"num_read_kb":925112,"num_write":236451,"num_write_kb":945774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'231371","reported_seq":463349,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:13.941175+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:02:13.941175+0000","last_peered":"2023-10-14T01:02:13.941175+0000","last_clean":"2023-10-14T01:02:13.941175+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:02:13.941175+0000","last_undegraded":"2023-10-14T01:02:13.941175+0000","last_fullsized":"2023-10-14T01:02:13.941175+0000","mapping_epoch":13,"log_start":"131'221304","ondisk_log_start":"131'221304","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3216224387,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":232461,"num_read_kb":912833,"num_write":230766,"num_write_kb":922986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"134'260587","reported_seq":522118,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:12.837281+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:02:12.837281+0000","last_peered":"2023-10-14T01:02:12.837281+0000","last_clean":"2023-10-14T01:02:12.837281+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:02:12.837281+0000","last_undegraded":"2023-10-14T01:02:12.837281+0000","last_fullsized":"2023-10-14T01:02:12.837281+0000","mapping_epoch":13,"log_start":"131'250507","ondisk_log_start":"131'250507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":48,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3617001791,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":262375,"num_read_kb":1024570,"num_write":259038,"num_write_kb":1036005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"134'251518","reported_seq":499337,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:11.795755+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:02:11.795755+0000","last_peered":"2023-10-14T01:02:11.795755+0000","last_clean":"2023-10-14T01:02:11.795755+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:02:11.795755+0000","last_undegraded":"2023-10-14T01:02:11.795755+0000","last_fullsized":"2023-10-14T01:02:11.795755+0000","mapping_epoch":13,"log_start":"131'241507","ondisk_log_start":"131'241507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":14,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3394330893,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":249615,"num_read_kb":962460,"num_write":244292,"num_write_kb":976954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"134'255271","reported_seq":509343,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:11.779088+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:02:11.779088+0000","last_peered":"2023-10-14T01:02:11.779088+0000","last_clean":"2023-10-14T01:02:11.779088+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:02:11.779088+0000","last_undegraded":"2023-10-14T01:02:11.779088+0000","last_fullsized":"2023-10-14T01:02:11.779088+0000","mapping_epoch":13,"log_start":"131'245209","ondisk_log_start":"131'245209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":96,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3441557753,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":256175,"num_read_kb":982596,"num_write":246407,"num_write_kb":985354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"134'250830","reported_seq":496068,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:13.922759+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:02:13.922759+0000","last_peered":"2023-10-14T01:02:13.922759+0000","last_clean":"2023-10-14T01:02:13.922759+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:02:13.922759+0000","last_undegraded":"2023-10-14T01:02:13.922759+0000","last_fullsized":"2023-10-14T01:02:13.922759+0000","mapping_epoch":13,"log_start":"131'240807","ondisk_log_start":"131'240807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3380895928,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":246343,"num_read_kb":959847,"num_write":242706,"num_write_kb":970711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":337,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:58.396639+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:01:58.396639+0000","last_peered":"2023-10-14T01:01:58.396639+0000","last_clean":"2023-10-14T01:01:58.396639+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:01:58.396639+0000","last_undegraded":"2023-10-14T01:01:58.396639+0000","last_fullsized":"2023-10-14T01:01:58.396639+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27212589368,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":1993174,"num_read_kb":7728394,"num_write":1952624,"num_write_kb":7809025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15992541184,"data_stored":20304020080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80376,"ondisk_log_size":80376,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739161,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3751512,"kb_used_data":2933324,"kb_used_omap":212,"kb_used_meta":817963,"kb_avail":90620328,"statfs":{"total":96636764160,"available":92795215872,"internally_reserved":0,"allocated":3003723776,"data_stored":3812950858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":837594356},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.74299999999999999,"15min":0.85799999999999998},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.14699999999999999},"max":{"1min":5.165,"5min":6.7270000000000003,"15min":10.803000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.026,"15min":0.71499999999999997},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.9420000000000002,"5min":10.847,"15min":10.847},"last":0.23000000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:02:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.2589999999999999,"15min":0.94099999999999995},"min":{"1min":0.24099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":8.1839999999999993,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":4.2469999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.91000000000000003,"15min":1.0700000000000001},"min":{"1min":0.249,"5min":0.19,"15min":0.19},"max":{"1min":2.903,"5min":9.6549999999999994,"15min":16.736999999999998},"last":0.20699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739151,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9847256,"kb_used_data":7810704,"kb_used_omap":563,"kb_used_meta":2035980,"kb_avail":84524584,"statfs":{"total":96636764160,"available":86553174016,"internally_reserved":0,"allocated":7998160896,"data_stored":10153688036,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":577123,"internal_metadata":2084843933},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,2,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:02:11 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.86699999999999999,"15min":0.92700000000000005},"min":{"1min":0.22,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.206,"5min":10.891,"15min":13.109},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.82299999999999995,"15min":0.92000000000000004},"min":{"1min":0.25700000000000001,"5min":0.192,"15min":0.192},"max":{"1min":3.3109999999999999,"5min":10.856,"15min":13.076000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:01:21 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":1.3049999999999999,"15min":1.139},"min":{"1min":0.254,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":3.552,"5min":14.522,"15min":14.522},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.2070000000000001,"15min":0.92500000000000004},"min":{"1min":0.24099999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.2670000000000003,"5min":14.537000000000001,"15min":14.537000000000001},"last":0.36899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6164696,"kb_used_data":4876172,"kb_used_omap":355,"kb_used_meta":1288156,"kb_avail":88207144,"statfs":{"total":96636764160,"available":90324115456,"internally_reserved":0,"allocated":4993200128,"data_stored":6338754854,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1319071952},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:02:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.0189999999999999,"15min":1.0589999999999999},"min":{"1min":0.19,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":5.2939999999999996,"5min":8.5239999999999991,"15min":12.855},"last":3.7949999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63700000000000001,"15min":1.167},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":3.972,"5min":5.1269999999999998,"15min":16.655000000000001},"last":6.944}]},{"osd":2,"last update":"Sat Oct 14 01:02:06 2023","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.399,"15min":1.2769999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":8.9019999999999992,"5min":9.1999999999999993,"15min":19.809999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.371,"5min":0.70499999999999996,"15min":0.84499999999999997},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":8.8249999999999993,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4992036864,"data_stored":6337803898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7997566976,"data_stored":10153306424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3002937344,"data_stored":3812909758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:02:16.989 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.990 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.990 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.990 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.991 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.991 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.991 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.992 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:16.992 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:02:17.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:17.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:17.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:17.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:17.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:17.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:17.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:17.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:17.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:17.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:18.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:17.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:18.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:18.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:18.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:18.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:18.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:18.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:18.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:18.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:18.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:18.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:18.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:18.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:18.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:19.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.189+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:19.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.201+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:19.219 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:19.219 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:19.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:19.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:19.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:19.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:19.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:19.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:19.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:19.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:19.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:19.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:19.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:19.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:19.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.865+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:19.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:19.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:02:19.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:19.897+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:21.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:21.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:21.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:21.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:21.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:21.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:21.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:21.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:21.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:21.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:21.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:21.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:24.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:02:24.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:02:24.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:24.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:24.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:24.401 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:24.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:24.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:24.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:24.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:24.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:25.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:25.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:25.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:25.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:25.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:25.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:28.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:28.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:28.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:28.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:28.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:28.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:28.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:28.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:28.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:28.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:28.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:28.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:29.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:29.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:29.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:29.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:29.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:29.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:29.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:29.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:30.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:30.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:30.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:30.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:30.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:30.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:30.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:30.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:36.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:36.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:36.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:36.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:36.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:36.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:36.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.288+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:36.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:02:36.805 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:36.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:36.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:36.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:36.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:36.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:36.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:36.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:36.993 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:02:37.614 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:02:37.615 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:02:37.685 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2126,"stamp":"2023-10-14T01:02:37.480416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27250662232,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2002290,"num_read_kb":7764331,"num_write":1962013,"num_write_kb":7848453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80452,"ondisk_log_size":80452,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19666420,"kb_used_data":15691288,"kb_used_omap":1133,"kb_used_meta":3973970,"kb_avail":263449100,"statfs":{"total":289910292480,"available":269771878400,"internally_reserved":0,"allocated":16067878912,"data_stored":20378187860,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4069345490},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,5,1,8,10,19,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19746816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4680,"num_read_kb":18664,"num_write":4858,"num_write_kb":19432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002126"},"pg_stats":[{"pgid":"2.7","version":"134'262931","reported_seq":516492,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:34.491502+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:02:34.491502+0000","last_peered":"2023-10-14T01:02:34.491502+0000","last_clean":"2023-10-14T01:02:34.491502+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:02:34.491502+0000","last_undegraded":"2023-10-14T01:02:34.491502+0000","last_fullsized":"2023-10-14T01:02:34.491502+0000","mapping_epoch":13,"log_start":"132'252902","ondisk_log_start":"132'252902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3445330358,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":255393,"num_read_kb":984767,"num_write":247426,"num_write_kb":989462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'263228","reported_seq":517672,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:34.484610+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:02:34.484610+0000","last_peered":"2023-10-14T01:02:34.484610+0000","last_clean":"2023-10-14T01:02:34.484610+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:02:34.484610+0000","last_undegraded":"2023-10-14T01:02:34.484610+0000","last_fullsized":"2023-10-14T01:02:34.484610+0000","mapping_epoch":13,"log_start":"132'253209","ondisk_log_start":"132'253209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3439563120,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":257874,"num_read_kb":984757,"num_write":247729,"num_write_kb":990543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'238191","reported_seq":473852,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:34.477809+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:02:34.477809+0000","last_peered":"2023-10-14T01:02:34.477809+0000","last_clean":"2023-10-14T01:02:34.477809+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:02:34.477809+0000","last_undegraded":"2023-10-14T01:02:34.477809+0000","last_fullsized":"2023-10-14T01:02:34.477809+0000","mapping_epoch":13,"log_start":"131'228105","ondisk_log_start":"131'228105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3291165074,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":236104,"num_read_kb":929192,"num_write":237573,"num_write_kb":950262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'232415","reported_seq":465390,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:34.458355+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:02:34.458355+0000","last_peered":"2023-10-14T01:02:34.458355+0000","last_clean":"2023-10-14T01:02:34.458355+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:02:34.458355+0000","last_undegraded":"2023-10-14T01:02:34.458355+0000","last_fullsized":"2023-10-14T01:02:34.458355+0000","mapping_epoch":13,"log_start":"131'222404","ondisk_log_start":"131'222404","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3220476035,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":233457,"num_read_kb":916813,"num_write":231810,"num_write_kb":927162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"134'261743","reported_seq":524457,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:32.945016+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:02:32.945016+0000","last_peered":"2023-10-14T01:02:32.945016+0000","last_clean":"2023-10-14T01:02:32.945016+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:02:32.945016+0000","last_undegraded":"2023-10-14T01:02:32.945016+0000","last_fullsized":"2023-10-14T01:02:32.945016+0000","mapping_epoch":13,"log_start":"132'251707","ondisk_log_start":"132'251707","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":70,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 38s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3621658943,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":263549,"num_read_kb":1029238,"num_write":260179,"num_write_kb":1040569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"134'252869","reported_seq":501974,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:37.148343+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:02:37.148343+0000","last_peered":"2023-10-14T01:02:37.148343+0000","last_clean":"2023-10-14T01:02:37.148343+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:02:37.148343+0000","last_undegraded":"2023-10-14T01:02:37.148343+0000","last_fullsized":"2023-10-14T01:02:37.148343+0000","mapping_epoch":13,"log_start":"132'242807","ondisk_log_start":"132'242807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":98,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 37s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3399553293,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":250869,"num_read_kb":967448,"num_write":245582,"num_write_kb":982114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"134'256640","reported_seq":512085,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:37.149270+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:02:37.149270+0000","last_peered":"2023-10-14T01:02:37.149270+0000","last_clean":"2023-10-14T01:02:37.149270+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:02:37.149270+0000","last_undegraded":"2023-10-14T01:02:37.149270+0000","last_fullsized":"2023-10-14T01:02:37.149270+0000","mapping_epoch":13,"log_start":"132'246609","ondisk_log_start":"132'246609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":136,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 40s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3446931705,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":257516,"num_read_kb":987944,"num_write":247733,"num_write_kb":990658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"134'251958","reported_seq":498263,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:34.481410+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:02:34.481410+0000","last_peered":"2023-10-14T01:02:34.481410+0000","last_clean":"2023-10-14T01:02:34.481410+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:02:34.481410+0000","last_undegraded":"2023-10-14T01:02:34.481410+0000","last_fullsized":"2023-10-14T01:02:34.481410+0000","mapping_epoch":13,"log_start":"132'241907","ondisk_log_start":"132'241907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3385393336,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":247402,"num_read_kb":964063,"num_write":243812,"num_write_kb":975135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":337,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:58.396639+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:01:58.396639+0000","last_peered":"2023-10-14T01:01:58.396639+0000","last_clean":"2023-10-14T01:01:58.396639+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:01:58.396639+0000","last_undegraded":"2023-10-14T01:01:58.396639+0000","last_fullsized":"2023-10-14T01:01:58.396639+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27250071864,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2002164,"num_read_kb":7764222,"num_write":1961844,"num_write_kb":7845905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16066822144,"data_stored":20378301040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80325,"ondisk_log_size":80325,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739166,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3821600,"kb_used_data":2949272,"kb_used_omap":212,"kb_used_meta":872107,"kb_avail":90550240,"statfs":{"total":96636764160,"available":92723445760,"internally_reserved":0,"allocated":3020054528,"data_stored":3829281610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":893037812},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.74299999999999999,"15min":0.85799999999999998},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.14699999999999999},"max":{"1min":5.165,"5min":6.7270000000000003,"15min":10.803000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.026,"15min":0.71499999999999997},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.9420000000000002,"5min":10.847,"15min":10.847},"last":4.2720000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:02:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.2589999999999999,"15min":0.94099999999999995},"min":{"1min":0.24099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":8.1839999999999993,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.91000000000000003,"15min":1.0700000000000001},"min":{"1min":0.249,"5min":0.19,"15min":0.19},"max":{"1min":2.903,"5min":9.6549999999999994,"15min":16.736999999999998},"last":4.3140000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739155,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9561092,"kb_used_data":7845048,"kb_used_omap":565,"kb_used_meta":1715466,"kb_avail":84810748,"statfs":{"total":96636764160,"available":86846205952,"internally_reserved":0,"allocated":8033329152,"data_stored":10188856292,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1756637966},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,1,3,5,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:02:11 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.86699999999999999,"15min":0.92700000000000005},"min":{"1min":0.22,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.206,"5min":10.891,"15min":13.109},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.82299999999999995,"15min":0.92000000000000004},"min":{"1min":0.25700000000000001,"5min":0.192,"15min":0.192},"max":{"1min":3.3109999999999999,"5min":10.856,"15min":13.076000000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.97099999999999997,"15min":1.1180000000000001},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":5.4690000000000003,"5min":6.8470000000000004,"15min":14.522},"last":0.25},{"interface":"front","average":{"1min":0.59599999999999997,"5min":1.012,"15min":0.90100000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.1299999999999999,"5min":10.571,"15min":14.537000000000001},"last":0.23699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739164,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6283728,"kb_used_data":4896968,"kb_used_omap":355,"kb_used_meta":1386396,"kb_avail":88088112,"statfs":{"total":96636764160,"available":90202226688,"internally_reserved":0,"allocated":5014495232,"data_stored":6360049958,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1419669712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,0,2,3,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:02:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.0189999999999999,"15min":1.0589999999999999},"min":{"1min":0.19,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":5.2939999999999996,"5min":8.5239999999999991,"15min":12.855},"last":0.373},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63700000000000001,"15min":1.167},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":3.972,"5min":5.1269999999999998,"15min":16.655000000000001},"last":0.218}]},{"osd":2,"last update":"Sat Oct 14 01:02:06 2023","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.399,"15min":1.2769999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":8.9019999999999992,"5min":9.1999999999999993,"15min":19.809999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.371,"5min":0.70499999999999996,"15min":0.84499999999999997},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":8.8249999999999993,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.23100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5014298624,"data_stored":6360065658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8033304576,"data_stored":10189044024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3019218944,"data_stored":3829191358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:02:37.687 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.687 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.688 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.689 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.689 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.690 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.690 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.690 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:37.691 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:02:38.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:38.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:38.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:38.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:38.043 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:38.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:02:38.044 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:38.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:02:38.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:38.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:38.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:38.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:38.081 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:38.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:38.081 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:38.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:38.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:38.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:38.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:38.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:41.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:41.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:41.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:41.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:41.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:41.424+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:41.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:41.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:41.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:41.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:41.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:41.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:42.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:42.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:42.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:42.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:42.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:42.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:42.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:42.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:42.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:42.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:42.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:43.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:43.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:43.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:43.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:43.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:43.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:43.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:43.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:43.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:43.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:43.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:43.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:43.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:43.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:44.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:44.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:44.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:44.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:44.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:44.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:44.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:44.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:44.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:44.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:44.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:44.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:44.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:44.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:47.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:47.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:47.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:47.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:47.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:47.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:47.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:47.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:47.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:47.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:49.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:49.120+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:49.802 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:49.800+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:49.815 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:49.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:49.820 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:49.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:51.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:51.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:51.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:51.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:51.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:51.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:51.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:51.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:51.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:52.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:52.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:52.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:52.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:52.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:52.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:53.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:53.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:02:53.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:53.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:53.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:53.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:02:53.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:53.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:56.176 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:56.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:56.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:56.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:56.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:02:56.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:02:56.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:02:56.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:02:56.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:02:56.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:02:56.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:02:56.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:02:56.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:02:57.692 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:02:58.768 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:02:58.769 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:02:58.830 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2136,"stamp":"2023-10-14T01:02:57.487925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27286072152,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2010717,"num_read_kb":7797951,"num_write":1970707,"num_write_kb":7883229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80695,"ondisk_log_size":80695,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19956988,"kb_used_data":15761316,"kb_used_omap":1133,"kb_used_meta":4194514,"kb_avail":263158532,"statfs":{"total":289910292480,"available":269474336768,"internally_reserved":0,"allocated":16139587584,"data_stored":20449896532,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4295182546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,10,10,13,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19890176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4845,"num_read_kb":19328,"num_write":4890,"num_write_kb":19560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006167"},"pg_stats":[{"pgid":"2.7","version":"134'264077","reported_seq":518716,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:54.983493+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:02:54.983493+0000","last_peered":"2023-10-14T01:02:54.983493+0000","last_clean":"2023-10-14T01:02:54.983493+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:02:54.983493+0000","last_undegraded":"2023-10-14T01:02:54.983493+0000","last_fullsized":"2023-10-14T01:02:54.983493+0000","mapping_epoch":13,"log_start":"132'254002","ondisk_log_start":"132'254002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3449823670,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":256458,"num_read_kb":989015,"num_write":248528,"num_write_kb":993870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"134'264359","reported_seq":519891,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:54.984241+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:02:54.984241+0000","last_peered":"2023-10-14T01:02:54.984241+0000","last_clean":"2023-10-14T01:02:54.984241+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:02:54.984241+0000","last_undegraded":"2023-10-14T01:02:54.984241+0000","last_fullsized":"2023-10-14T01:02:54.984241+0000","mapping_epoch":13,"log_start":"132'254309","ondisk_log_start":"132'254309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3443974512,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":258949,"num_read_kb":989045,"num_write":248816,"num_write_kb":994891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"134'239247","reported_seq":475928,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:54.999153+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:02:54.999153+0000","last_peered":"2023-10-14T01:02:54.999153+0000","last_clean":"2023-10-14T01:02:54.999153+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:02:54.999153+0000","last_undegraded":"2023-10-14T01:02:54.999153+0000","last_fullsized":"2023-10-14T01:02:54.999153+0000","mapping_epoch":13,"log_start":"132'229205","ondisk_log_start":"132'229205","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3295490450,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":237124,"num_read_kb":933268,"num_write":238629,"num_write_kb":954486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"134'233479","reported_seq":467464,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:54.967301+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:02:54.967301+0000","last_peered":"2023-10-14T01:02:54.967301+0000","last_clean":"2023-10-14T01:02:54.967301+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:02:54.967301+0000","last_undegraded":"2023-10-14T01:02:54.967301+0000","last_fullsized":"2023-10-14T01:02:54.967301+0000","mapping_epoch":13,"log_start":"132'223404","ondisk_log_start":"132'223404","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3224817795,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":234469,"num_read_kb":920853,"num_write":232874,"num_write_kb":931418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"134'262898","reported_seq":526744,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:53.036803+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:02:53.036803+0000","last_peered":"2023-10-14T01:02:53.036803+0000","last_clean":"2023-10-14T01:02:53.036803+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:02:53.036803+0000","last_undegraded":"2023-10-14T01:02:53.036803+0000","last_fullsized":"2023-10-14T01:02:53.036803+0000","mapping_epoch":13,"log_start":"132'252807","ondisk_log_start":"132'252807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":97,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 58s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3626287423,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":264671,"num_read_kb":1033710,"num_write":261316,"num_write_kb":1045117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"134'254004","reported_seq":504203,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:57.440697+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:02:57.440697+0000","last_peered":"2023-10-14T01:02:57.440697+0000","last_clean":"2023-10-14T01:02:57.440697+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:02:57.440697+0000","last_undegraded":"2023-10-14T01:02:57.440697+0000","last_fullsized":"2023-10-14T01:02:57.440697+0000","mapping_epoch":13,"log_start":"132'243907","ondisk_log_start":"132'243907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":130,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 57s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3403976973,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":251943,"num_read_kb":971736,"num_write":246669,"num_write_kb":986462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"134'257762","reported_seq":514284,"reported_epoch":134,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:02:57.395047+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:02:57.395047+0000","last_peered":"2023-10-14T01:02:57.395047+0000","last_clean":"2023-10-14T01:02:57.395047+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:02:57.395047+0000","last_undegraded":"2023-10-14T01:02:57.395047+0000","last_fullsized":"2023-10-14T01:02:57.395047+0000","mapping_epoch":13,"log_start":"132'247709","ondisk_log_start":"132'247709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":201,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 61s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3451212025,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":258566,"num_read_kb":992124,"num_write":248782,"num_write_kb":994854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"134'253092","reported_seq":500414,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:02:54.986341+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:02:54.986341+0000","last_peered":"2023-10-14T01:02:54.986341+0000","last_clean":"2023-10-14T01:02:54.986341+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:02:54.986341+0000","last_undegraded":"2023-10-14T01:02:54.986341+0000","last_fullsized":"2023-10-14T01:02:54.986341+0000","mapping_epoch":13,"log_start":"132'243007","ondisk_log_start":"132'243007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3389898936,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":248411,"num_read_kb":968091,"num_write":244924,"num_write_kb":979583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":337,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-14T01:01:58.396639+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:01:58.396639+0000","last_peered":"2023-10-14T01:01:58.396639+0000","last_clean":"2023-10-14T01:01:58.396639+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:01:58.396639+0000","last_undegraded":"2023-10-14T01:01:58.396639+0000","last_fullsized":"2023-10-14T01:01:58.396639+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27285481784,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2010591,"num_read_kb":7797842,"num_write":1970538,"num_write_kb":7880681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16137781248,"data_stored":20449260144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80568,"ondisk_log_size":80568,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3866292,"kb_used_data":2962028,"kb_used_omap":212,"kb_used_meta":904043,"kb_avail":90505548,"statfs":{"total":96636764160,"available":92677681152,"internally_reserved":0,"allocated":3033116672,"data_stored":3842343754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":925740276},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,4,3,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:02:00 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.74299999999999999,"15min":0.85799999999999998},"min":{"1min":0.19600000000000001,"5min":0.189,"15min":0.14699999999999999},"max":{"1min":5.165,"5min":6.7270000000000003,"15min":10.803000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.026,"15min":0.71499999999999997},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.9420000000000002,"5min":10.847,"15min":10.847},"last":0.248}]},{"osd":1,"last update":"Sat Oct 14 01:02:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.2589999999999999,"15min":0.94099999999999995},"min":{"1min":0.24099999999999999,"5min":0.191,"15min":0.191},"max":{"1min":8.1839999999999993,"5min":9.6199999999999992,"15min":9.6199999999999992},"last":0.747},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.91000000000000003,"15min":1.0700000000000001},"min":{"1min":0.249,"5min":0.19,"15min":0.19},"max":{"1min":2.903,"5min":9.6549999999999994,"15min":16.736999999999998},"last":0.67400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9725904,"kb_used_data":7880968,"kb_used_omap":565,"kb_used_meta":1844362,"kb_avail":84645936,"statfs":{"total":96636764160,"available":86677438464,"internally_reserved":0,"allocated":8070111232,"data_stored":10225638372,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1888627470},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,7,5,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:02:11 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.86699999999999999,"15min":0.92700000000000005},"min":{"1min":0.22,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.206,"5min":10.891,"15min":13.109},"last":0.629},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.82299999999999995,"15min":0.92000000000000004},"min":{"1min":0.25700000000000001,"5min":0.192,"15min":0.192},"max":{"1min":3.3109999999999999,"5min":10.856,"15min":13.076000000000001},"last":0.26900000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.97099999999999997,"15min":1.1180000000000001},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":5.4690000000000003,"5min":6.8470000000000004,"15min":14.522},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":1.012,"15min":0.90100000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.1299999999999999,"5min":10.571,"15min":14.537000000000001},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739168,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6364792,"kb_used_data":4918320,"kb_used_omap":355,"kb_used_meta":1446108,"kb_avail":88007048,"statfs":{"total":96636764160,"available":90119217152,"internally_reserved":0,"allocated":5036359680,"data_stored":6381914406,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1480814800},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:02:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.0189999999999999,"15min":1.0589999999999999},"min":{"1min":0.19,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":5.2939999999999996,"5min":8.5239999999999991,"15min":12.855},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.63700000000000001,"15min":1.167},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":3.972,"5min":5.1269999999999998,"15min":16.655000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:02:06 2023","interfaces":[{"interface":"back","average":{"1min":1.821,"5min":1.399,"15min":1.2769999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":8.9019999999999992,"5min":9.1999999999999993,"15min":19.809999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.371,"5min":0.70499999999999996,"15min":0.84499999999999997},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":8.8249999999999993,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.36399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5036097536,"data_stored":6381864570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8069140480,"data_stored":10224879928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3032543232,"data_stored":3842515646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:02:58.832 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.833 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.833 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.834 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.834 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.835 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.835 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.835 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:02:58.836 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:02:58.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:58.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:58.964 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:58.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:58.964 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:58.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:02:59.517 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:59.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:02:59.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:59.552+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:02:59.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:59.552+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:02:59.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:02:59.552+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:00.060 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:00.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:00.363 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:00.360+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:00.372 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:00.372+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:00.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:03:00.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:00.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:00.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:00.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:00.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:00.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:00.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:00.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:00.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:00.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:00.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:01.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:01.480+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:03:01.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:01.500+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:03.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:03.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:03.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:03.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:03.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:03.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:03.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.968+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:03:03.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:03.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:03.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:03.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:03.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:04.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:04.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:04.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:04.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:04.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:04.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:06.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:06.312+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:03:06.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:06.324+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:03:06.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:06.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:06.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:06.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:06.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:06.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:06.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:06.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:06.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:06.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:06.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:06.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:06.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:06.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:10.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:10.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:10.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:10.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:10.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:10.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:10.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:10.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:10.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:10.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:10.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:10.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:10.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:10.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:12.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:12.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:12.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:12.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:12.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:12.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:12.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:12.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:12.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:12.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:12.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:12.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:12.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:12.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:13.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:13.068+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:13.718 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:13.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:13.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:13.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:13.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:13.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:14.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:14.716+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:15.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:15.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:15.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:15.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:15.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:15.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:17.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:17.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:17.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:17.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:17.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:17.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:18.836 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:03:19.604 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:03:19.604 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:03:19.661 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2148,"stamp":"2023-10-14T01:03:19.492130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27323190104,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2019351,"num_read_kb":7832351,"num_write":1979821,"num_write_kb":7919685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80509,"ondisk_log_size":80509,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20309472,"kb_used_data":15836416,"kb_used_omap":1133,"kb_used_meta":4471890,"kb_avail":262806048,"statfs":{"total":289910292480,"available":269113393152,"internally_reserved":0,"allocated":16216489984,"data_stored":20526776495,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4579215570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,21,5,19,26],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18579456,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4388,"num_read_kb":17496,"num_write":4561,"num_write_kb":18244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001866"},"pg_stats":[{"pgid":"2.7","version":"135'265254","reported_seq":520958,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:15.533635+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:03:15.533635+0000","last_peered":"2023-10-14T01:03:15.533635+0000","last_clean":"2023-10-14T01:03:15.533635+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:03:15.533635+0000","last_undegraded":"2023-10-14T01:03:15.533635+0000","last_fullsized":"2023-10-14T01:03:15.533635+0000","mapping_epoch":13,"log_start":"132'255202","ondisk_log_start":"132'255202","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3454378422,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":257505,"num_read_kb":993195,"num_write":249656,"num_write_kb":998382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'265540","reported_seq":522125,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:15.560137+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:03:15.560137+0000","last_peered":"2023-10-14T01:03:15.560137+0000","last_clean":"2023-10-14T01:03:15.560137+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:03:15.560137+0000","last_undegraded":"2023-10-14T01:03:15.560137+0000","last_fullsized":"2023-10-14T01:03:15.560137+0000","mapping_epoch":13,"log_start":"132'255509","ondisk_log_start":"132'255509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3448566128,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":259982,"num_read_kb":993157,"num_write":249948,"num_write_kb":999419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'240282","reported_seq":477949,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:15.521094+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:03:15.521094+0000","last_peered":"2023-10-14T01:03:15.521094+0000","last_clean":"2023-10-14T01:03:15.521094+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:03:15.521094+0000","last_undegraded":"2023-10-14T01:03:15.521094+0000","last_fullsized":"2023-10-14T01:03:15.521094+0000","mapping_epoch":13,"log_start":"132'230205","ondisk_log_start":"132'230205","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3299701138,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":238110,"num_read_kb":937204,"num_write":239664,"num_write_kb":958626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'234510","reported_seq":469464,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:15.558155+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:03:15.558155+0000","last_peered":"2023-10-14T01:03:15.558155+0000","last_clean":"2023-10-14T01:03:15.558155+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:03:15.558155+0000","last_undegraded":"2023-10-14T01:03:15.558155+0000","last_fullsized":"2023-10-14T01:03:15.558155+0000","mapping_epoch":13,"log_start":"132'224504","ondisk_log_start":"132'224504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3229036675,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":235437,"num_read_kb":924705,"num_write":233905,"num_write_kb":935542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"135'264363","reported_seq":529649,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:18.338628+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:03:18.338628+0000","last_peered":"2023-10-14T01:03:18.338628+0000","last_clean":"2023-10-14T01:03:18.338628+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:03:18.338628+0000","last_undegraded":"2023-10-14T01:03:18.338628+0000","last_fullsized":"2023-10-14T01:03:18.338628+0000","mapping_epoch":13,"log_start":"132'254307","ondisk_log_start":"132'254307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":155,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 83s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3632103743,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":266088,"num_read_kb":1039354,"num_write":262744,"num_write_kb":1050829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"135'255229","reported_seq":506565,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:18.914550+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:03:18.914550+0000","last_peered":"2023-10-14T01:03:18.914550+0000","last_clean":"2023-10-14T01:03:18.914550+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:03:18.914550+0000","last_undegraded":"2023-10-14T01:03:18.914550+0000","last_fullsized":"2023-10-14T01:03:18.914550+0000","mapping_epoch":13,"log_start":"132'245207","ondisk_log_start":"132'245207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":182,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 78s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3408724237,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":253055,"num_read_kb":976164,"num_write":247824,"num_write_kb":991082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"135'258998","reported_seq":516634,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:18.849270+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:03:18.849270+0000","last_peered":"2023-10-14T01:03:18.849270+0000","last_clean":"2023-10-14T01:03:18.849270+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:03:18.849270+0000","last_undegraded":"2023-10-14T01:03:18.849270+0000","last_fullsized":"2023-10-14T01:03:18.849270+0000","mapping_epoch":13,"log_start":"132'248909","ondisk_log_start":"132'248909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":236,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 82s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3455955193,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":259652,"num_read_kb":996444,"num_write":249947,"num_write_kb":999514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"135'254156","reported_seq":502472,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:15.533696+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:03:15.533696+0000","last_peered":"2023-10-14T01:03:15.533696+0000","last_clean":"2023-10-14T01:03:15.533696+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:03:15.533696+0000","last_undegraded":"2023-10-14T01:03:15.533696+0000","last_fullsized":"2023-10-14T01:03:15.533696+0000","mapping_epoch":13,"log_start":"132'244107","ondisk_log_start":"132'244107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3394134200,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":249396,"num_read_kb":972019,"num_write":245964,"num_write_kb":983743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":338,"reported_epoch":134,"state":"active+clean","last_fresh":"2023-10-14T01:01:59.326797+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:01:59.326797+0000","last_peered":"2023-10-14T01:01:59.326797+0000","last_clean":"2023-10-14T01:01:59.326797+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:01:59.326797+0000","last_undegraded":"2023-10-14T01:01:59.326797+0000","last_fullsized":"2023-10-14T01:01:59.326797+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27322599736,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2019225,"num_read_kb":7832242,"num_write":1979652,"num_write_kb":7917137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16215064576,"data_stored":20526543472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80382,"ondisk_log_size":80382,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739175,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3952456,"kb_used_data":2975996,"kb_used_omap":212,"kb_used_meta":976235,"kb_avail":90419384,"statfs":{"total":96636764160,"available":92589449216,"internally_reserved":0,"allocated":3047419904,"data_stored":3856639507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":999664884},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.65800000000000003,"15min":0.878},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.14699999999999999},"max":{"1min":5.6970000000000001,"5min":5.875,"15min":10.803000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0469999999999999,"15min":0.76300000000000001},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.7619999999999996,"5min":10.847,"15min":10.847},"last":0.53400000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.417,"15min":0.97099999999999997},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.191},"max":{"1min":14.103999999999999,"5min":14.103999999999999,"15min":14.103999999999999},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.006,"15min":1.0640000000000001},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":4.9420000000000002,"5min":9.6549999999999994,"15min":16.736999999999998},"last":0.57399999999999995}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9876488,"kb_used_data":7914940,"kb_used_omap":565,"kb_used_meta":1960970,"kb_avail":84495352,"statfs":{"total":96636764160,"available":86523240448,"internally_reserved":0,"allocated":8104898560,"data_stored":10260418221,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":2008034062},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,15,1,14,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.78400000000000003,"15min":0.91000000000000003},"min":{"1min":0.20999999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.753,"5min":10.891,"15min":13.109},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.73199999999999998,"15min":0.89300000000000002},"min":{"1min":0.26900000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.78000000000000003,"5min":10.856,"15min":13.076000000000001},"last":0.248}]},{"osd":2,"last update":"Sat Oct 14 01:02:23 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.97099999999999997,"15min":1.1180000000000001},"min":{"1min":0.29099999999999998,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":5.4690000000000003,"5min":6.8470000000000004,"15min":14.522},"last":0.313},{"interface":"front","average":{"1min":0.59599999999999997,"5min":1.012,"15min":0.90100000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.1299999999999999,"5min":10.571,"15min":14.537000000000001},"last":0.32500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739173,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6480528,"kb_used_data":4945480,"kb_used_omap":355,"kb_used_meta":1534684,"kb_avail":87891312,"statfs":{"total":96636764160,"available":90000703488,"internally_reserved":0,"allocated":5064171520,"data_stored":6409718767,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1571516624},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:03:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.1299999999999999,"15min":1.0980000000000001},"min":{"1min":0.248,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":6.4539999999999997,"5min":8.5239999999999991,"15min":12.855},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.63800000000000001,"15min":1.1279999999999999},"min":{"1min":0.218,"5min":0.192,"15min":0.192},"max":{"1min":6.944,"5min":6.944,"15min":16.655000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:03:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.4059999999999999,"15min":1.238},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.3899999999999997,"5min":9.1999999999999993,"15min":10.053000000000001},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.746,"15min":0.85199999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.3499999999999996,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5063790592,"data_stored":6409557626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8104837120,"data_stored":10260576568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3046436864,"data_stored":3856409278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:03:19.663 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.663 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.663 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.664 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.664 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.665 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.665 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.665 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:19.666 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:03:20.088 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:20.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:20.088 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:20.087+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:20.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:20.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:21.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:21.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:21.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:21.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:21.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:21.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:21.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:21.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:23.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:03:23.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:23.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:23.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:23.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:23.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:23.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:23.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:23.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:25.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:25.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:25.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:25.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:25.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:25.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:25.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:25.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:26.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:26.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:26.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:26.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:27.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:27.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:27.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:27.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:27.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:27.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:28.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:03:28.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:28.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:28.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:28.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:28.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:28.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:03:28.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:28.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:28.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:28.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:28.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:28.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:28.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:28.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:29.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:29.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:29.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:29.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:29.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:29.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:29.525 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:29.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:03:29.526 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:29.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:03:29.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:29.539+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:29.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:29.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:29.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:29.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:29.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:29.547+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:31.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:31.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:32.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:32.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:32.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:32.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:32.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:32.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:32.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:32.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:32.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:32.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:32.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:32.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:32.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:32.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:32.948 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:32.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:32.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:34.652 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:34.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:03:34.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:34.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:03:34.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:34.655+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:34.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:34.655+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:34.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:34.655+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:36.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:36.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:36.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:36.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:36.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:36.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:36.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:36.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:36.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:36.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:36.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:36.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:38.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:38.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:38.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:38.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:38.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:38.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:38.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:38.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:38.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:38.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:39.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:03:39.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:39.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:40.600 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:03:40.602 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:03:40.640 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2158,"stamp":"2023-10-14T01:03:39.499110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27358124888,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2027555,"num_read_kb":7865111,"num_write":1988362,"num_write_kb":7953849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80622,"ondisk_log_size":80622,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20606148,"kb_used_data":15906728,"kb_used_omap":1133,"kb_used_meta":4698258,"kb_avail":262509372,"statfs":{"total":289910292480,"available":268809596928,"internally_reserved":0,"allocated":16288489472,"data_stored":20598775983,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4811016402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,6,10,7,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20684800,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4936,"num_read_kb":19712,"num_write":5068,"num_write_kb":20272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001925"},"pg_stats":[{"pgid":"2.7","version":"135'266385","reported_seq":523154,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.156396+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:03:36.156396+0000","last_peered":"2023-10-14T01:03:36.156396+0000","last_clean":"2023-10-14T01:03:36.156396+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:03:36.156396+0000","last_undegraded":"2023-10-14T01:03:36.156396+0000","last_fullsized":"2023-10-14T01:03:36.156396+0000","mapping_epoch":13,"log_start":"132'256302","ondisk_log_start":"132'256302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3458810294,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":258553,"num_read_kb":997387,"num_write":250739,"num_write_kb":1002714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'266692","reported_seq":524283,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.120617+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:03:36.120617+0000","last_peered":"2023-10-14T01:03:36.120617+0000","last_clean":"2023-10-14T01:03:36.120617+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:03:36.120617+0000","last_undegraded":"2023-10-14T01:03:36.120617+0000","last_fullsized":"2023-10-14T01:03:36.120617+0000","mapping_epoch":13,"log_start":"132'256609","ondisk_log_start":"132'256609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3453084016,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":260973,"num_read_kb":997113,"num_write":251052,"num_write_kb":1003835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'241337","reported_seq":479995,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.097866+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:03:36.097866+0000","last_peered":"2023-10-14T01:03:36.097866+0000","last_clean":"2023-10-14T01:03:36.097866+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:03:36.097866+0000","last_undegraded":"2023-10-14T01:03:36.097866+0000","last_fullsized":"2023-10-14T01:03:36.097866+0000","mapping_epoch":13,"log_start":"132'231305","ondisk_log_start":"132'231305","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3304010130,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":239101,"num_read_kb":941156,"num_write":240719,"num_write_kb":962846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'235540","reported_seq":471482,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.141836+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:03:36.141836+0000","last_peered":"2023-10-14T01:03:36.141836+0000","last_clean":"2023-10-14T01:03:36.141836+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:03:36.141836+0000","last_undegraded":"2023-10-14T01:03:36.141836+0000","last_fullsized":"2023-10-14T01:03:36.141836+0000","mapping_epoch":13,"log_start":"132'225504","ondisk_log_start":"132'225504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3233255555,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":236425,"num_read_kb":928649,"num_write":234935,"num_write_kb":939662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"135'265491","reported_seq":531864,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:38.555169+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:03:38.555169+0000","last_peered":"2023-10-14T01:03:38.555169+0000","last_clean":"2023-10-14T01:03:38.555169+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:03:38.555169+0000","last_undegraded":"2023-10-14T01:03:38.555169+0000","last_fullsized":"2023-10-14T01:03:38.555169+0000","mapping_epoch":13,"log_start":"133'255407","ondisk_log_start":"133'255407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":211,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 103s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3636551999,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":267168,"num_read_kb":1043666,"num_write":263836,"num_write_kb":1055197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"135'256365","reported_seq":508740,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:39.137858+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:03:39.137858+0000","last_peered":"2023-10-14T01:03:39.137858+0000","last_clean":"2023-10-14T01:03:39.137858+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:03:39.137858+0000","last_undegraded":"2023-10-14T01:03:39.137858+0000","last_fullsized":"2023-10-14T01:03:39.137858+0000","mapping_epoch":13,"log_start":"132'246307","ondisk_log_start":"132'246307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":220,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 99s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3413209357,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":254069,"num_read_kb":980208,"num_write":248912,"num_write_kb":995434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"135'260078","reported_seq":518835,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:39.088989+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:03:39.088989+0000","last_peered":"2023-10-14T01:03:39.088989+0000","last_clean":"2023-10-14T01:03:39.088989+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:03:39.088989+0000","last_undegraded":"2023-10-14T01:03:39.088989+0000","last_fullsized":"2023-10-14T01:03:39.088989+0000","mapping_epoch":13,"log_start":"132'250010","ondisk_log_start":"132'250010","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":294,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 102s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3460079865,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":260739,"num_read_kb":1000788,"num_write":250958,"num_write_kb":1003558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"135'255258","reported_seq":504586,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.129824+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:03:36.129824+0000","last_peered":"2023-10-14T01:03:36.129824+0000","last_clean":"2023-10-14T01:03:36.129824+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:03:36.129824+0000","last_undegraded":"2023-10-14T01:03:36.129824+0000","last_fullsized":"2023-10-14T01:03:36.129824+0000","mapping_epoch":13,"log_start":"132'245207","ondisk_log_start":"132'245207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3398533304,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":250401,"num_read_kb":976035,"num_write":247042,"num_write_kb":988055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":339,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:14.383969+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:03:14.383969+0000","last_peered":"2023-10-14T01:03:14.383969+0000","last_clean":"2023-10-14T01:03:14.383969+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:03:14.383969+0000","last_undegraded":"2023-10-14T01:03:14.383969+0000","last_fullsized":"2023-10-14T01:03:14.383969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27357534520,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2027429,"num_read_kb":7865002,"num_write":1988193,"num_write_kb":7951301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16285106176,"data_stored":20596585072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80495,"ondisk_log_size":80495,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739179,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3997060,"kb_used_data":2988668,"kb_used_omap":212,"kb_used_meta":1008171,"kb_avail":90374780,"statfs":{"total":96636764160,"available":92543774720,"internally_reserved":0,"allocated":3060396032,"data_stored":3869615635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1032367348},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.65800000000000003,"15min":0.878},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.14699999999999999},"max":{"1min":5.6970000000000001,"5min":5.875,"15min":10.803000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0469999999999999,"15min":0.76300000000000001},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.7619999999999996,"5min":10.847,"15min":10.847},"last":0.34300000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.417,"15min":0.97099999999999997},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.191},"max":{"1min":14.103999999999999,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.006,"15min":1.0640000000000001},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":4.9420000000000002,"5min":9.6549999999999994,"15min":16.736999999999998},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739167,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10012320,"kb_used_data":7950744,"kb_used_omap":565,"kb_used_meta":2061002,"kb_avail":84359520,"statfs":{"total":96636764160,"available":86384148480,"internally_reserved":0,"allocated":8141561856,"data_stored":10297081517,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":2110466830},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,6,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.78400000000000003,"15min":0.91000000000000003},"min":{"1min":0.20999999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.753,"5min":10.891,"15min":13.109},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.73199999999999998,"15min":0.89300000000000002},"min":{"1min":0.26900000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.78000000000000003,"5min":10.856,"15min":13.076000000000001},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.89200000000000002,"15min":1.099},"min":{"1min":0.25,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":4.734,"5min":6.4729999999999999,"15min":14.522},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.80000000000000004,"15min":0.92100000000000004},"min":{"1min":0.20499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":14.537000000000001},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739177,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6596768,"kb_used_data":4967316,"kb_used_omap":355,"kb_used_meta":1629084,"kb_avail":87775072,"statfs":{"total":96636764160,"available":89881673728,"internally_reserved":0,"allocated":5086531584,"data_stored":6432078831,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1668182224},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:03:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.1299999999999999,"15min":1.0980000000000001},"min":{"1min":0.248,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":6.4539999999999997,"5min":8.5239999999999991,"15min":12.855},"last":0.438},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.63800000000000001,"15min":1.1279999999999999},"min":{"1min":0.218,"5min":0.192,"15min":0.192},"max":{"1min":6.944,"5min":6.944,"15min":16.655000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:03:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.4059999999999999,"15min":1.238},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.3899999999999997,"5min":9.1999999999999993,"15min":10.053000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.746,"15min":0.85199999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.3499999999999996,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.47299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5085343744,"data_stored":6431110778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8140451840,"data_stored":10296191288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3059310592,"data_stored":3869283006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:03:40.642 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.643 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.643 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.644 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.644 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.645 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.646 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.646 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:03:40.646 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:03:40.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:03:40.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:40.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:40.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:40.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:40.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:40.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:40.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:40.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:40.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:40.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:41.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:03:41.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:03:41.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:03:41.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:41.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:41.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:41.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:41.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:41.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:41.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:41.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:41.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:41.859 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:41.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:03:41.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:41.887+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:03:41.898 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:41.895+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:41.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:41.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:41.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:41.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:41.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:41.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:43.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:43.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:43.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:43.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:43.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:43.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:43.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:43.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:43.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:43.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:43.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:43.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:44.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:44.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:44.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:44.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:44.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:44.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:44.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:44.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:44.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:44.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:44.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:44.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:44.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:45.393 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:03:45.391+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:03:45.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:45.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:45.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:45.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:45.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:45.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:45.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:45.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:45.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:45.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:47.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:47.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:03:47.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:47.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:03:47.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:47.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:03:47.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:47.055+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:47.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:47.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:47.118 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:47.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:47.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:47.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:48.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:48.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:48.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:48.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:48.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:48.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:48.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:48.103+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:48.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:48.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:48.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:48.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:48.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:48.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:49.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:49.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:03:49.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:49.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:49.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:49.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:49.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:49.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:49.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:49.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:49.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:49.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:49.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:49.975+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:03:49.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:49.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:49.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:49.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:50.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:50.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:50.617 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:50.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:50.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:50.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:50.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:50.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:50.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:50.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:50.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:51.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:03:51.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:03:51.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:03:51.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:51.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:51.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:51.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:51.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:51.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:51.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:51.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:51.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:51.659+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:03:51.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:51.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:03:51.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:51.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:03:51.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:03:51.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:03:53.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:53.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:53.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:53.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:53.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:53.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:53.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:53.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:53.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:53.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:53.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:53.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:53.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:53.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:55.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:55.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:55.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:55.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:55.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:55.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:56.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:56.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:03:56.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:56.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:03:56.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:56.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:03:56.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:56.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:56.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:56.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:56.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:56.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:56.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:56.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:03:59.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:59.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:03:59.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:59.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:03:59.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:59.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:03:59.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:03:59.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:00.647 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:04:01.204 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:04:01.205 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:04:01.253 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2168,"stamp":"2023-10-14T01:03:59.502403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27392428888,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2036295,"num_read_kb":7899975,"num_write":1996726,"num_write_kb":7987305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80509,"ondisk_log_size":80509,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19815824,"kb_used_data":15973044,"kb_used_omap":1133,"kb_used_meta":3841618,"kb_avail":263299696,"statfs":{"total":289910292480,"available":269618888704,"internally_reserved":0,"allocated":16356397056,"data_stored":20666683567,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":3933817042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,12,17,14,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20504576,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5191,"num_read_kb":20696,"num_write":5015,"num_write_kb":20060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002001"},"pg_stats":[{"pgid":"2.7","version":"135'267571","reported_seq":525456,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:56.645650+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:03:56.645650+0000","last_peered":"2023-10-14T01:03:56.645650+0000","last_clean":"2023-10-14T01:03:56.645650+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:03:56.645650+0000","last_undegraded":"2023-10-14T01:03:56.645650+0000","last_fullsized":"2023-10-14T01:03:56.645650+0000","mapping_epoch":13,"log_start":"133'257502","ondisk_log_start":"133'257502","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3463360950,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":259653,"num_read_kb":1001775,"num_write":251849,"num_write_kb":1007154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'267792","reported_seq":526517,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:56.639540+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:03:56.639540+0000","last_peered":"2023-10-14T01:03:56.639540+0000","last_clean":"2023-10-14T01:03:56.639540+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:03:56.639540+0000","last_undegraded":"2023-10-14T01:03:56.639540+0000","last_fullsized":"2023-10-14T01:03:56.639540+0000","mapping_epoch":13,"log_start":"133'257709","ondisk_log_start":"133'257709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3457298800,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":262091,"num_read_kb":1001577,"num_write":252076,"num_write_kb":1007931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'242339","reported_seq":482069,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:56.621649+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:03:56.621649+0000","last_peered":"2023-10-14T01:03:56.621649+0000","last_clean":"2023-10-14T01:03:56.621649+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:03:56.621649+0000","last_undegraded":"2023-10-14T01:03:56.621649+0000","last_fullsized":"2023-10-14T01:03:56.621649+0000","mapping_epoch":13,"log_start":"132'232305","ondisk_log_start":"132'232305","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3308122514,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":240172,"num_read_kb":945416,"num_write":241721,"num_write_kb":966854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'236573","reported_seq":473532,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:56.629599+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:03:56.629599+0000","last_peered":"2023-10-14T01:03:56.629599+0000","last_clean":"2023-10-14T01:03:56.629599+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:03:56.629599+0000","last_undegraded":"2023-10-14T01:03:56.629599+0000","last_fullsized":"2023-10-14T01:03:56.629599+0000","mapping_epoch":13,"log_start":"132'226504","ondisk_log_start":"132'226504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3237503107,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":237440,"num_read_kb":932709,"num_write":235968,"num_write_kb":943794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"135'266654","reported_seq":534271,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:58.862441+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:03:58.862441+0000","last_peered":"2023-10-14T01:03:58.862441+0000","last_clean":"2023-10-14T01:03:58.862441+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:03:58.862441+0000","last_undegraded":"2023-10-14T01:03:58.862441+0000","last_fullsized":"2023-10-14T01:03:58.862441+0000","mapping_epoch":13,"log_start":"133'256607","ondisk_log_start":"133'256607","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":241,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 124s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3641225535,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":268396,"num_read_kb":1048558,"num_write":264980,"num_write_kb":1059773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"135'257434","reported_seq":510904,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:59.355615+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:03:59.355615+0000","last_peered":"2023-10-14T01:03:59.355615+0000","last_clean":"2023-10-14T01:03:59.355615+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:03:59.355615+0000","last_undegraded":"2023-10-14T01:03:59.355615+0000","last_fullsized":"2023-10-14T01:03:59.355615+0000","mapping_epoch":13,"log_start":"133'247407","ondisk_log_start":"133'247407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":278,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 119s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3417383181,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":255136,"num_read_kb":984468,"num_write":249929,"num_write_kb":999502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"135'261153","reported_seq":521018,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:03:59.356749+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:03:59.356749+0000","last_peered":"2023-10-14T01:03:59.356749+0000","last_clean":"2023-10-14T01:03:59.356749+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:03:59.356749+0000","last_undegraded":"2023-10-14T01:03:59.356749+0000","last_fullsized":"2023-10-14T01:03:59.356749+0000","mapping_epoch":13,"log_start":"133'251110","ondisk_log_start":"133'251110","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":365,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 123s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3464245497,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":261818,"num_read_kb":1005096,"num_write":251972,"num_write_kb":1007614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"135'256317","reported_seq":506713,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:56.618849+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:03:56.618849+0000","last_peered":"2023-10-14T01:03:56.618849+0000","last_clean":"2023-10-14T01:03:56.618849+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:03:56.618849+0000","last_undegraded":"2023-10-14T01:03:56.618849+0000","last_fullsized":"2023-10-14T01:03:56.618849+0000","mapping_epoch":13,"log_start":"133'246307","ondisk_log_start":"133'246307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3402698936,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":251463,"num_read_kb":980267,"num_write":248062,"num_write_kb":992135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":339,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:14.383969+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:03:14.383969+0000","last_peered":"2023-10-14T01:03:14.383969+0000","last_clean":"2023-10-14T01:03:14.383969+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:03:14.383969+0000","last_undegraded":"2023-10-14T01:03:14.383969+0000","last_fullsized":"2023-10-14T01:03:14.383969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27391838520,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2036169,"num_read_kb":7899866,"num_write":1996557,"num_write_kb":7984757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16353693696,"data_stored":20665172592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80382,"ondisk_log_size":80382,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739183,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4059120,"kb_used_data":3000744,"kb_used_omap":212,"kb_used_meta":1058155,"kb_avail":90312720,"statfs":{"total":96636764160,"available":92480225280,"internally_reserved":0,"allocated":3072761856,"data_stored":3881981459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1083550964},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,6,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:03:00 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.65800000000000003,"15min":0.878},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.14699999999999999},"max":{"1min":5.6970000000000001,"5min":5.875,"15min":10.803000000000001},"last":0.504},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0469999999999999,"15min":0.76300000000000001},"min":{"1min":0.23000000000000001,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":5.7619999999999996,"5min":10.847,"15min":10.847},"last":0.53700000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:03:00 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.417,"15min":0.97099999999999997},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.191},"max":{"1min":14.103999999999999,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.006,"15min":1.0640000000000001},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.19},"max":{"1min":4.9420000000000002,"5min":9.6549999999999994,"15min":16.736999999999998},"last":0.31900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739171,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9097140,"kb_used_data":7983852,"kb_used_omap":565,"kb_used_meta":1112714,"kb_avail":85274700,"statfs":{"total":96636764160,"available":87321292800,"internally_reserved":0,"allocated":8175464448,"data_stored":10330984109,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1139419918},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,7,8,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:03:12 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.78400000000000003,"15min":0.91000000000000003},"min":{"1min":0.20999999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.753,"5min":10.891,"15min":13.109},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.73199999999999998,"15min":0.89300000000000002},"min":{"1min":0.26900000000000002,"5min":0.192,"15min":0.192},"max":{"1min":0.78000000000000003,"5min":10.856,"15min":13.076000000000001},"last":0.497}]},{"osd":2,"last update":"Sat Oct 14 01:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.89200000000000002,"15min":1.099},"min":{"1min":0.25,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":4.734,"5min":6.4729999999999999,"15min":14.522},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.80000000000000004,"15min":0.92100000000000004},"min":{"1min":0.20499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":14.537000000000001},"last":0.22500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739181,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6659564,"kb_used_data":4988448,"kb_used_omap":355,"kb_used_meta":1670748,"kb_avail":87712276,"statfs":{"total":96636764160,"available":89817370624,"internally_reserved":0,"allocated":5108170752,"data_stored":6453717999,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1710846160},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:03:06 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.1299999999999999,"15min":1.0980000000000001},"min":{"1min":0.248,"5min":0.17999999999999999,"15min":0.16800000000000001},"max":{"1min":6.4539999999999997,"5min":8.5239999999999991,"15min":12.855},"last":0.248},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.63800000000000001,"15min":1.1279999999999999},"min":{"1min":0.218,"5min":0.192,"15min":0.192},"max":{"1min":6.944,"5min":6.944,"15min":16.655000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:03:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.4059999999999999,"15min":1.238},"min":{"1min":0.20999999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":4.3899999999999997,"5min":9.1999999999999993,"15min":10.053000000000001},"last":5.1520000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.746,"15min":0.85199999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.3499999999999996,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.20699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5107118080,"data_stored":6452885114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8174796800,"data_stored":10330536248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3071778816,"data_stored":3881751230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:04:01.255 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.255 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.256 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.256 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.256 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.257 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.257 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.258 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:01.258 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:04:01.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:01.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:01.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:01.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:01.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:01.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:03.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:03.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:03.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:03.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:03.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:03.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:03.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:03.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:03.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:03.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:03.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:03.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:06.414 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:06.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:06.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:06.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:06.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:06.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:06.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:06.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:08.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:08.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:04:08.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:08.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:08.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:08.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:09.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:09.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:09.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:09.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:09.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:09.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:09.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:09.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:09.609 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:04:09.609 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:04:09.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:04:09.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:09.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:09.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:09.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:09.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:09.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:09.602+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:10.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:10.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:10.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:10.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:10.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:10.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:10.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:10.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:04:10.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:04:10.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:10.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:10.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:10.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:10.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:10.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:10.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:10.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:10.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:10.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:10.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:10.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:10.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:10.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:10.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:10.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:10.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:10.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:10.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:10.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:10.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:10.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:10.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:11.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:11.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:11.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:11.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:11.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:11.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:11.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:11.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:11.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:11.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:11.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:11.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:11.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:11.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:14.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:14.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:14.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:14.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:14.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:14.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:14.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:14.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:16.525 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:16.522+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:16.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:16.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:16.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:16.530+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:17.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:04:17.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:04:17.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:17.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:17.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:17.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:17.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:17.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:17.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:17.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:17.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:18.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:18.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:18.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:18.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:18.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:18.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:18.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:18.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:18.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:18.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:18.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:18.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:21.259 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:04:22.140 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:04:22.141 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:04:22.173 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2179,"stamp":"2023-10-14T01:04:21.506071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27427642200,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2044829,"num_read_kb":7934035,"num_write":2005272,"num_write_kb":8021489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80639,"ondisk_log_size":80639,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20104052,"kb_used_data":16040284,"kb_used_omap":1133,"kb_used_meta":4062610,"kb_avail":263011468,"statfs":{"total":289910292480,"available":269323743232,"internally_reserved":0,"allocated":16425250816,"data_stored":20735537327,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4160112850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,6,41,19,15,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19677184,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4790,"num_read_kb":19096,"num_write":4772,"num_write_kb":19088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002046"},"pg_stats":[{"pgid":"2.7","version":"135'268746","reported_seq":527763,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:04:17.215969+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:04:17.215969+0000","last_peered":"2023-10-14T01:04:17.215969+0000","last_clean":"2023-10-14T01:04:17.215969+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:04:17.215969+0000","last_undegraded":"2023-10-14T01:04:17.215969+0000","last_fullsized":"2023-10-14T01:04:17.215969+0000","mapping_epoch":13,"log_start":"133'258702","ondisk_log_start":"133'258702","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3467960758,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":260768,"num_read_kb":1006223,"num_write":252968,"num_write_kb":1011630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"135'268974","reported_seq":528767,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:04:17.209211+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:04:17.209211+0000","last_peered":"2023-10-14T01:04:17.209211+0000","last_clean":"2023-10-14T01:04:17.209211+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:04:17.209211+0000","last_undegraded":"2023-10-14T01:04:17.209211+0000","last_fullsized":"2023-10-14T01:04:17.209211+0000","mapping_epoch":13,"log_start":"133'258909","ondisk_log_start":"133'258909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3461935472,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":263143,"num_read_kb":1005773,"num_write":253206,"num_write_kb":1012451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"135'243359","reported_seq":484115,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:04:17.215723+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:04:17.215723+0000","last_peered":"2023-10-14T01:04:17.215723+0000","last_clean":"2023-10-14T01:04:17.215723+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:04:17.215723+0000","last_undegraded":"2023-10-14T01:04:17.215723+0000","last_fullsized":"2023-10-14T01:04:17.215723+0000","mapping_epoch":13,"log_start":"133'233305","ondisk_log_start":"133'233305","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3312378258,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":241204,"num_read_kb":949532,"num_write":242741,"num_write_kb":970934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"135'237541","reported_seq":475528,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:04:17.154720+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:04:17.154720+0000","last_peered":"2023-10-14T01:04:17.154720+0000","last_clean":"2023-10-14T01:04:17.154720+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:04:17.154720+0000","last_undegraded":"2023-10-14T01:04:17.154720+0000","last_fullsized":"2023-10-14T01:04:17.154720+0000","mapping_epoch":13,"log_start":"133'227504","ondisk_log_start":"133'227504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3241537667,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":238468,"num_read_kb":936809,"num_write":236936,"num_write_kb":947666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"135'267791","reported_seq":536509,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:04:18.938397+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:04:18.938397+0000","last_peered":"2023-10-14T01:04:18.938397+0000","last_clean":"2023-10-14T01:04:18.938397+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:04:18.938397+0000","last_undegraded":"2023-10-14T01:04:18.938397+0000","last_fullsized":"2023-10-14T01:04:18.938397+0000","mapping_epoch":13,"log_start":"133'257707","ondisk_log_start":"133'257707","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":277,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 144s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3645813055,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":269486,"num_read_kb":1052914,"num_write":266094,"num_write_kb":1064229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"135'258562","reported_seq":513142,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:04:19.620931+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:04:19.620931+0000","last_peered":"2023-10-14T01:04:19.620931+0000","last_clean":"2023-10-14T01:04:19.620931+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:04:19.620931+0000","last_undegraded":"2023-10-14T01:04:19.620931+0000","last_fullsized":"2023-10-14T01:04:19.620931+0000","mapping_epoch":13,"log_start":"133'248507","ondisk_log_start":"133'248507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":345,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 139s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3421708557,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":256219,"num_read_kb":988788,"num_write":250991,"num_write_kb":1003750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"135'262290","reported_seq":523259,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:04:19.560872+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:04:19.560872+0000","last_peered":"2023-10-14T01:04:19.560872+0000","last_clean":"2023-10-14T01:04:19.560872+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:04:19.560872+0000","last_undegraded":"2023-10-14T01:04:19.560872+0000","last_fullsized":"2023-10-14T01:04:19.560872+0000","mapping_epoch":13,"log_start":"133'252210","ondisk_log_start":"133'252210","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":414,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 143s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3468689657,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":262895,"num_read_kb":1009396,"num_write":253048,"num_write_kb":1011918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"135'257400","reported_seq":508865,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:04:17.159875+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:04:17.159875+0000","last_peered":"2023-10-14T01:04:17.159875+0000","last_clean":"2023-10-14T01:04:17.159875+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:04:17.159875+0000","last_undegraded":"2023-10-14T01:04:17.159875+0000","last_fullsized":"2023-10-14T01:04:17.159875+0000","mapping_epoch":13,"log_start":"133'247307","ondisk_log_start":"133'247307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3407028408,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":252520,"num_read_kb":984491,"num_write":249119,"num_write_kb":996363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":339,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:14.383969+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:03:14.383969+0000","last_peered":"2023-10-14T01:03:14.383969+0000","last_clean":"2023-10-14T01:03:14.383969+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:03:14.383969+0000","last_undegraded":"2023-10-14T01:03:14.383969+0000","last_fullsized":"2023-10-14T01:03:14.383969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27427051832,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2044703,"num_read_kb":7933926,"num_write":2005103,"num_write_kb":8018941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16423456768,"data_stored":20734935664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80512,"ondisk_log_size":80512,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739187,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4102392,"kb_used_data":3013488,"kb_used_omap":212,"kb_used_meta":1088683,"kb_avail":90269448,"statfs":{"total":96636764160,"available":92435914752,"internally_reserved":0,"allocated":3085811712,"data_stored":3895031315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1114811636},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,36,5,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.65800000000000003,"15min":0.85999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.14699999999999999},"max":{"1min":2.7770000000000001,"5min":5.875,"15min":10.803000000000001},"last":5.5419999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.93200000000000005,"15min":0.75700000000000001},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.71,"5min":10.847,"15min":10.847},"last":0.19900000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.407,"15min":0.93000000000000005},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":2.3940000000000001,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.077,"15min":1.1120000000000001},"min":{"1min":0.21099999999999999,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":6.4850000000000003,"5min":9.6549999999999994,"15min":16.736999999999998},"last":0.30299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739175,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9236452,"kb_used_data":8017500,"kb_used_omap":565,"kb_used_meta":1218378,"kb_avail":85135388,"statfs":{"total":96636764160,"available":87178637312,"internally_reserved":0,"allocated":8209920000,"data_stored":10365439661,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1247619854},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,10,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.84799999999999998,"15min":0.90800000000000003},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.16800000000000001},"max":{"1min":4.6180000000000003,"5min":10.891,"15min":12.369999999999999},"last":4.2160000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76000000000000001,"15min":0.872},"min":{"1min":0.24199999999999999,"5min":0.22,"15min":0.192},"max":{"1min":4.6580000000000004,"5min":10.856,"15min":12.407},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:03:26 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.89200000000000002,"15min":1.099},"min":{"1min":0.25,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":4.734,"5min":6.4729999999999999,"15min":14.522},"last":4.25},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.80000000000000004,"15min":0.92100000000000004},"min":{"1min":0.20499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.0599999999999996,"5min":7.0599999999999996,"15min":14.537000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739185,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6765208,"kb_used_data":5009296,"kb_used_omap":355,"kb_used_meta":1755548,"kb_avail":87606632,"statfs":{"total":96636764160,"available":89709191168,"internally_reserved":0,"allocated":5129519104,"data_stored":6475066351,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1797681360},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:04:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.177,"15min":1.0880000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":5.226,"5min":8.5239999999999991,"15min":12.855},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.53500000000000003,"15min":1.0820000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.64600000000000002,"5min":6.944,"15min":16.655000000000001},"last":0.17499999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:04:08 2023","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.4119999999999999,"15min":1.2929999999999999},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.0999999999999996,"5min":9.1999999999999993,"15min":10.053000000000001},"last":6.2069999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.66500000000000004,"15min":0.81399999999999995},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47299999999999998,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5128601600,"data_stored":6474368634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8210022400,"data_stored":10365761848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3084832768,"data_stored":3894805182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:04:22.174 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.175 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.175 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.175 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.176 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.176 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.177 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.177 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:22.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:04:22.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:22.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:22.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:22.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:22.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:22.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:22.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:22.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:22.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:22.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:22.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:22.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:22.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:22.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:22.998 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:04:23.039 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-14T01:04:23.787 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-14T01:04:23.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:04:23.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:23.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:23.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:23.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:23.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:23.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:23.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:23.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:23.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:23.875 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:04:24.473 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-14T01:04:24.470+0000 7fe065ca5640 0 --2- 172.21.15.138:0/1224563387 >> v2:172.21.15.138:3300/0 conn(0x7fe058014470 0x7fe05801d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-14T01:04:24.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:24.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:24.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:24.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:24.903 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:04:24.933 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-14T01:04:25.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:25.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:25.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:25.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:25.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:25.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:25.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:25.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:25.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:25.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:25.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:25.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:25.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:25.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:25.660 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-14T01:04:25.732 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-10-14T01:04:26.476 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-10-14T01:04:26.550 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-10-14T01:04:27.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:27.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:27.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:27.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:27.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:27.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:27.296 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-10-14T01:04:27.349 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-14T01:04:27.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:04:27.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:27.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:27.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:27.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:27.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:27.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:27.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:27.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:27.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:27.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:27.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:27.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:27.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:27.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:28.277 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-14T01:04:28.333 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-14T01:04:29.167 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-14T01:04:29.212 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:04:30.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:30.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:30.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:30.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:30.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:30.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:30.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:30.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:30.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:30.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:30.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:30.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:04:30.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:30.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:30.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:30.762+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:30.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:30.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:30.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:30.770+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:30.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:30.770+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:30.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:30.770+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:31.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:31.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:31.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:31.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:31.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:31.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:31.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:31.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:31.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:31.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:33.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:33.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:33.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:33.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:33.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:33.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:33.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:33.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:33.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:36.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:36.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:04:36.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:36.270+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:36.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:36.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:36.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:36.318+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:36.331 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:36.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:36.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:36.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:36.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:36.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:38.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:38.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:38.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:38.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:38.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:38.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:38.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:38.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:38.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:38.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:38.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:39.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:04:39.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:39.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:39.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:39.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:39.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:39.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:39.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:39.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:39.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:39.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:39.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:39.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:39.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:39.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:39.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:39.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:39.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:39.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:39.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:39.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:39.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:42.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:42.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:42.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:42.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:42.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:42.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:42.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:42.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:42.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:42.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:42.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:42.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:42.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:42.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:43.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:43.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:43.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:43.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:43.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:43.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:43.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:43.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:43.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:43.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:43.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:43.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:45.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:45.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:46.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:46.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:46.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:46.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:46.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:46.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:46.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:46.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:46.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:46.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:46.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:47.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:47.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:47.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:47.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:47.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:47.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:47.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:47.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:47.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:47.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:47.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:47.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:48.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:48.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:48.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:48.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:48.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:48.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:48.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:48.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:49.213 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:04:49.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:49.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:50.018 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:04:50.018 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:04:50.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:50.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:04:50.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:50.078+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:50.087 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2194,"stamp":"2023-10-14T01:04:49.511398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27477289816,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2057361,"num_read_kb":7983959,"num_write":2017376,"num_write_kb":8069905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80536,"ondisk_log_size":80536,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20518588,"kb_used_data":16138148,"kb_used_omap":1133,"kb_used_meta":4379282,"kb_avail":262596932,"statfs":{"total":289910292480,"available":268899258368,"internally_reserved":0,"allocated":16525463552,"data_stored":20835727497,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4484384978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,5,12,13,17,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23961600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6088,"num_read_kb":24232,"num_write":5846,"num_write_kb":23384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002703"},"pg_stats":[{"pgid":"2.7","version":"136'270431","reported_seq":531170,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:04:48.013310+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:04:48.013310+0000","last_peered":"2023-10-14T01:04:48.013310+0000","last_clean":"2023-10-14T01:04:48.013310+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:04:48.013310+0000","last_undegraded":"2023-10-14T01:04:48.013310+0000","last_fullsized":"2023-10-14T01:04:48.013310+0000","mapping_epoch":13,"log_start":"133'260402","ondisk_log_start":"133'260402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3474510262,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":262465,"num_read_kb":1012987,"num_write":254556,"num_write_kb":1017982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'270680","reported_seq":532158,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:04:48.013850+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:04:48.013850+0000","last_peered":"2023-10-14T01:04:48.013850+0000","last_clean":"2023-10-14T01:04:48.013850+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:04:48.013850+0000","last_undegraded":"2023-10-14T01:04:48.013850+0000","last_fullsized":"2023-10-14T01:04:48.013850+0000","mapping_epoch":13,"log_start":"133'260609","ondisk_log_start":"133'260609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3468513648,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":264800,"num_read_kb":1012377,"num_write":254811,"num_write_kb":1018871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'244873","reported_seq":487221,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:04:48.012308+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:04:48.012308+0000","last_peered":"2023-10-14T01:04:48.012308+0000","last_clean":"2023-10-14T01:04:48.012308+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:04:48.012308+0000","last_undegraded":"2023-10-14T01:04:48.012308+0000","last_fullsized":"2023-10-14T01:04:48.012308+0000","mapping_epoch":13,"log_start":"133'234805","ondisk_log_start":"133'234805","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3318579602,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":242787,"num_read_kb":955836,"num_write":244255,"num_write_kb":976990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'239128","reported_seq":478629,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:04:47.989130+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:04:47.989130+0000","last_peered":"2023-10-14T01:04:47.989130+0000","last_clean":"2023-10-14T01:04:47.989130+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:04:47.989130+0000","last_undegraded":"2023-10-14T01:04:47.989130+0000","last_fullsized":"2023-10-14T01:04:47.989130+0000","mapping_epoch":13,"log_start":"133'229104","ondisk_log_start":"133'229104","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3248054403,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":239981,"num_read_kb":942825,"num_write":238523,"num_write_kb":954014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"136'269491","reported_seq":539970,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:04:49.292121+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:04:49.292121+0000","last_peered":"2023-10-14T01:04:49.292121+0000","last_clean":"2023-10-14T01:04:49.292121+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:04:49.292121+0000","last_undegraded":"2023-10-14T01:04:49.292121+0000","last_fullsized":"2023-10-14T01:04:49.292121+0000","mapping_epoch":13,"log_start":"133'259407","ondisk_log_start":"133'259407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":311,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 174s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3652669759,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":271226,"num_read_kb":1059834,"num_write":267767,"num_write_kb":1070921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"136'259868","reported_seq":515801,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:04:44.904676+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:04:44.904676+0000","last_peered":"2023-10-14T01:04:44.904676+0000","last_clean":"2023-10-14T01:04:44.904676+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:04:44.904676+0000","last_undegraded":"2023-10-14T01:04:44.904676+0000","last_fullsized":"2023-10-14T01:04:44.904676+0000","mapping_epoch":13,"log_start":"133'249807","ondisk_log_start":"133'249807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":413,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 164s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3426787597,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":257541,"num_read_kb":994064,"num_write":252237,"num_write_kb":1008734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"136'263663","reported_seq":526063,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:04:44.922022+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:04:44.922022+0000","last_peered":"2023-10-14T01:04:44.922022+0000","last_clean":"2023-10-14T01:04:44.922022+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:04:44.922022+0000","last_undegraded":"2023-10-14T01:04:44.922022+0000","last_fullsized":"2023-10-14T01:04:44.922022+0000","mapping_epoch":13,"log_start":"133'253610","ondisk_log_start":"133'253610","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":470,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 168s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3474084089,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":264285,"num_read_kb":1014936,"num_write":254362,"num_write_kb":1017174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"136'259026","reported_seq":512134,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:04:47.993975+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:04:47.993975+0000","last_peered":"2023-10-14T01:04:47.993975+0000","last_clean":"2023-10-14T01:04:47.993975+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:04:47.993975+0000","last_undegraded":"2023-10-14T01:04:47.993975+0000","last_fullsized":"2023-10-14T01:04:47.993975+0000","mapping_epoch":13,"log_start":"133'249007","ondisk_log_start":"133'249007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3413500088,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":254150,"num_read_kb":990991,"num_write":250696,"num_write_kb":1002671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":340,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.120433+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:03:36.120433+0000","last_peered":"2023-10-14T01:03:36.120433+0000","last_clean":"2023-10-14T01:03:36.120433+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:03:36.120433+0000","last_undegraded":"2023-10-14T01:03:36.120433+0000","last_fullsized":"2023-10-14T01:03:36.120433+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27476699448,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2057235,"num_read_kb":7983850,"num_write":2017207,"num_write_kb":8067357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16523476992,"data_stored":20834955888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80409,"ondisk_log_size":80409,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739192,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4171976,"kb_used_data":3028928,"kb_used_omap":212,"kb_used_meta":1142827,"kb_avail":90199864,"statfs":{"total":96636764160,"available":92364660736,"internally_reserved":0,"allocated":3101622272,"data_stored":3910834353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1170255092},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,0,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.65800000000000003,"15min":0.85999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.14699999999999999},"max":{"1min":2.7770000000000001,"5min":5.875,"15min":10.803000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.93200000000000005,"15min":0.75700000000000001},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":3.71,"5min":10.847,"15min":10.847},"last":0.23400000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:04:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.407,"15min":0.93000000000000005},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":2.3940000000000001,"5min":14.103999999999999,"15min":14.103999999999999},"last":6.7619999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.077,"15min":1.1120000000000001},"min":{"1min":0.21099999999999999,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":6.4850000000000003,"5min":9.6549999999999994,"15min":16.736999999999998},"last":0.40799999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739181,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9437716,"kb_used_data":8068812,"kb_used_omap":565,"kb_used_meta":1368330,"kb_avail":84934124,"statfs":{"total":96636764160,"available":86972542976,"internally_reserved":0,"allocated":8262463488,"data_stored":10417975627,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1401170702},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,9,10,10,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.84799999999999998,"15min":0.90800000000000003},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.16800000000000001},"max":{"1min":4.6180000000000003,"5min":10.891,"15min":12.369999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76000000000000001,"15min":0.872},"min":{"1min":0.24199999999999999,"5min":0.22,"15min":0.192},"max":{"1min":4.6580000000000004,"5min":10.856,"15min":12.407},"last":0.42299999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:04:30 2023","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.92200000000000004,"15min":1.1120000000000001},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":4.6920000000000002,"5min":6.4729999999999999,"15min":14.522},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.80900000000000005,"15min":0.92800000000000005},"min":{"1min":0.216,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":3.988,"5min":7.0599999999999996,"15min":14.537000000000001},"last":0.27500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739191,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6908896,"kb_used_data":5040408,"kb_used_omap":355,"kb_used_meta":1868124,"kb_avail":87462944,"statfs":{"total":96636764160,"available":89562054656,"internally_reserved":0,"allocated":5161377792,"data_stored":6506917517,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1912959184},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:04:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.177,"15min":1.0880000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":5.226,"5min":8.5239999999999991,"15min":12.855},"last":3.3319999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.53500000000000003,"15min":1.0820000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.64600000000000002,"5min":6.944,"15min":16.655000000000001},"last":3.2930000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:04:08 2023","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.4119999999999999,"15min":1.2929999999999999},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.0999999999999996,"5min":9.1999999999999993,"15min":10.053000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.66500000000000004,"15min":0.81399999999999995},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47299999999999998,"5min":8.8249999999999993,"15min":15.451000000000001},"last":3.3639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5160857600,"data_stored":6506624634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8262045696,"data_stored":10417785144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3100573696,"data_stored":3910546110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:04:50.089 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.089 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.090 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.090 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.091 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.091 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.091 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.092 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:04:50.092 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:04:50.093 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:50.086+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:50.120 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:50.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:50.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:50.140 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:50.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:50.141 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:50.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:50.141 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:50.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:50.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:50.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:50.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:50.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:50.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:50.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:50.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:50.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:50.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:50.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:50.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:50.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:51.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:51.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:04:51.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:51.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:51.122 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:51.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:53.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:53.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:04:53.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:53.650+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:53.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:53.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:53.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:53.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:53.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:53.658+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:54.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:54.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:54.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:54.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:54.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:54.546+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:54.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:54.554+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:54.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:54.554+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:54.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:54.554+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:54.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:04:54.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:04:54.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:54.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:54.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:54.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:54.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:54.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:54.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:54.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:55.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:55.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:04:55.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:55.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:04:55.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:55.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:04:55.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:55.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:55.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:55.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:55.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:55.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:55.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:55.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:56.181 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:56.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:04:56.181 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:56.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:56.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:56.186+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:04:56.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:56.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:56.233 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:56.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:04:56.241 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:56.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:04:56.241 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:56.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:04:57.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:57.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:58.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:58.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:58.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:58.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:58.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:58.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:04:58.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:58.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:04:58.400 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:58.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:04:58.431 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:04:58.425+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:04:59.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:59.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:04:59.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:59.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:04:59.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:59.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:04:59.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:04:59.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:04.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:04.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:04.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:04.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:04.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:04.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:04.247 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:04.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:04.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:04.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:04.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:04.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:04.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:04.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:05.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:05.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:05.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:05.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:05.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:05.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:05.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:05.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:05.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:05.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:05.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:05.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:06.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:06.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:06.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:06.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:06.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:06.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:06.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:06.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:06.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:06.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:06.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:07.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:07.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:07.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:07.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:07.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:07.429+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:07.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:07.461+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:07.476 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:07.469+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:07.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:07.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:07.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:07.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:07.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:07.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:07.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:07.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:07.909+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:10.093 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:05:10.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:10.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:10.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:10.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:10.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:10.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:10.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:10.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:10.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:10.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:10.832 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:05:10.833 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:05:10.900 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2204,"stamp":"2023-10-14T01:05:09.517307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27511778136,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2066065,"num_read_kb":8018611,"num_write":2025806,"num_write_kb":8103625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80407,"ondisk_log_size":80407,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20810292,"kb_used_data":16204440,"kb_used_omap":1133,"kb_used_meta":4604690,"kb_avail":262305228,"statfs":{"total":289910292480,"available":268600553472,"internally_reserved":0,"allocated":16593346560,"data_stored":20903610505,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4715202770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,3,8,9,18,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23584768,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6006,"num_read_kb":23896,"num_write":5772,"num_write_kb":23088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004616"},"pg_stats":[{"pgid":"2.7","version":"136'271621","reported_seq":533473,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:08.434266+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:05:08.434266+0000","last_peered":"2023-10-14T01:05:08.434266+0000","last_clean":"2023-10-14T01:05:08.434266+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:05:08.434266+0000","last_undegraded":"2023-10-14T01:05:08.434266+0000","last_fullsized":"2023-10-14T01:05:08.434266+0000","mapping_epoch":13,"log_start":"134'261602","ondisk_log_start":"134'261602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3479101878,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":263558,"num_read_kb":1017343,"num_write":255678,"num_write_kb":1022470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'271770","reported_seq":534363,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:08.449454+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:05:08.449454+0000","last_peered":"2023-10-14T01:05:08.449454+0000","last_clean":"2023-10-14T01:05:08.449454+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:05:08.449454+0000","last_undegraded":"2023-10-14T01:05:08.449454+0000","last_fullsized":"2023-10-14T01:05:08.449454+0000","mapping_epoch":13,"log_start":"134'261709","ondisk_log_start":"134'261709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3472675184,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":265903,"num_read_kb":1016777,"num_write":255833,"num_write_kb":1022959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'245931","reported_seq":489375,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:08.461841+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:05:08.461841+0000","last_peered":"2023-10-14T01:05:08.461841+0000","last_clean":"2023-10-14T01:05:08.461841+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:05:08.461841+0000","last_undegraded":"2023-10-14T01:05:08.461841+0000","last_fullsized":"2023-10-14T01:05:08.461841+0000","mapping_epoch":13,"log_start":"133'235905","ondisk_log_start":"133'235905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3322896786,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":243883,"num_read_kb":960208,"num_write":245313,"num_write_kb":981222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'240146","reported_seq":480696,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:08.456959+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:05:08.456959+0000","last_peered":"2023-10-14T01:05:08.456959+0000","last_clean":"2023-10-14T01:05:08.456959+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:05:08.456959+0000","last_undegraded":"2023-10-14T01:05:08.456959+0000","last_fullsized":"2023-10-14T01:05:08.456959+0000","mapping_epoch":13,"log_start":"133'230104","ondisk_log_start":"133'230104","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3252211843,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":241029,"num_read_kb":946985,"num_write":239541,"num_write_kb":958086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"136'270609","reported_seq":542229,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:09.450822+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:05:09.450822+0000","last_peered":"2023-10-14T01:05:09.450822+0000","last_clean":"2023-10-14T01:05:09.450822+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:05:09.450822+0000","last_undegraded":"2023-10-14T01:05:09.450822+0000","last_fullsized":"2023-10-14T01:05:09.450822+0000","mapping_epoch":13,"log_start":"134'260607","ondisk_log_start":"134'260607","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":352,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 194s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3657146687,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":272358,"num_read_kb":1064342,"num_write":268858,"num_write_kb":1075285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"136'260977","reported_seq":518040,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:05.356764+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:05:05.356764+0000","last_peered":"2023-10-14T01:05:05.356764+0000","last_clean":"2023-10-14T01:05:05.356764+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:05:05.356764+0000","last_undegraded":"2023-10-14T01:05:05.356764+0000","last_fullsized":"2023-10-14T01:05:05.356764+0000","mapping_epoch":13,"log_start":"134'250907","ondisk_log_start":"134'250907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":486,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 185s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3431055629,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":258647,"num_read_kb":998464,"num_write":253272,"num_write_kb":1012874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"136'264750","reported_seq":528248,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:05.249337+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:05:05.249337+0000","last_peered":"2023-10-14T01:05:05.249337+0000","last_clean":"2023-10-14T01:05:05.249337+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:05:05.249337+0000","last_undegraded":"2023-10-14T01:05:05.249337+0000","last_fullsized":"2023-10-14T01:05:05.249337+0000","mapping_epoch":13,"log_start":"134'254710","ondisk_log_start":"134'254710","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":523,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 189s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3478241529,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":265364,"num_read_kb":1019224,"num_write":255379,"num_write_kb":1021242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"136'260127","reported_seq":514291,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:08.437291+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:05:08.437291+0000","last_peered":"2023-10-14T01:05:08.437291+0000","last_clean":"2023-10-14T01:05:08.437291+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:05:08.437291+0000","last_undegraded":"2023-10-14T01:05:08.437291+0000","last_fullsized":"2023-10-14T01:05:08.437291+0000","mapping_epoch":13,"log_start":"133'250107","ondisk_log_start":"133'250107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3417858232,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":255197,"num_read_kb":995159,"num_write":251763,"num_write_kb":1006939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":340,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.120433+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:03:36.120433+0000","last_peered":"2023-10-14T01:03:36.120433+0000","last_clean":"2023-10-14T01:03:36.120433+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:03:36.120433+0000","last_undegraded":"2023-10-14T01:03:36.120433+0000","last_fullsized":"2023-10-14T01:03:36.120433+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27511187768,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2065939,"num_read_kb":8018502,"num_write":2025637,"num_write_kb":8101077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16592171008,"data_stored":20903649904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80280,"ondisk_log_size":80280,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739196,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4234296,"kb_used_data":3041260,"kb_used_omap":212,"kb_used_meta":1192811,"kb_avail":90137544,"statfs":{"total":96636764160,"available":92300845056,"internally_reserved":0,"allocated":3114250240,"data_stored":3923462321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1221438708},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:05:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.69899999999999995,"15min":0.84599999999999997},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.14699999999999999},"max":{"1min":5.5419999999999998,"5min":5.6970000000000001,"15min":10.803000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.93600000000000005,"15min":0.73599999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":4.306,"5min":10.847,"15min":10.847},"last":0.315}]},{"osd":1,"last update":"Sat Oct 14 01:05:03 2023","interfaces":[{"interface":"back","average":{"1min":2.6619999999999999,"5min":1.5800000000000001,"15min":1.0489999999999999},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.191},"max":{"1min":9.657,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.355,"5min":1.1040000000000001,"15min":1.1279999999999999},"min":{"1min":0.24299999999999999,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":5.2220000000000004,"5min":6.4850000000000003,"15min":16.736999999999998},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739185,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9587800,"kb_used_data":8101776,"kb_used_omap":565,"kb_used_meta":1485450,"kb_avail":84784040,"statfs":{"total":96636764160,"available":86818856960,"internally_reserved":0,"allocated":8296218624,"data_stored":10451730763,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1521101582},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,2,5,6,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:04:16 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.84799999999999998,"15min":0.90800000000000003},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.16800000000000001},"max":{"1min":4.6180000000000003,"5min":10.891,"15min":12.369999999999999},"last":0.31},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.76000000000000001,"15min":0.872},"min":{"1min":0.24199999999999999,"5min":0.22,"15min":0.192},"max":{"1min":4.6580000000000004,"5min":10.856,"15min":12.407},"last":0.34399999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:04:30 2023","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.92200000000000004,"15min":1.1120000000000001},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":4.6920000000000002,"5min":6.4729999999999999,"15min":14.522},"last":0.378},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.80900000000000005,"15min":0.92800000000000005},"min":{"1min":0.216,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":3.988,"5min":7.0599999999999996,"15min":14.537000000000001},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739195,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6988196,"kb_used_data":5061404,"kb_used_omap":355,"kb_used_meta":1926428,"kb_avail":87383644,"statfs":{"total":96636764160,"available":89480851456,"internally_reserved":0,"allocated":5182877696,"data_stored":6528417421,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":1972662480},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:04:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":1.177,"15min":1.0880000000000001},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":5.226,"5min":8.5239999999999991,"15min":12.855},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.53500000000000003,"15min":1.0820000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.64600000000000002,"5min":6.944,"15min":16.655000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:04:08 2023","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.4119999999999999,"15min":1.2929999999999999},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.0999999999999996,"5min":9.1999999999999993,"15min":10.053000000000001},"last":1.98},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.66500000000000004,"15min":0.81399999999999995},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47299999999999998,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5182283776,"data_stored":6528050810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8296587264,"data_stored":10452326712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3113299968,"data_stored":3923272382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:05:10.902 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.902 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.902 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.903 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.903 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.904 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.904 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.904 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:10.905 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:05:11.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:11.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:11.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:11.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:11.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:11.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:11.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:11.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:11.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:11.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:11.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:11.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:11.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:11.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:14.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:14.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:14.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:14.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:14.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:14.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:14.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:14.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:14.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:14.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:14.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:14.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:14.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:15.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:15.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:15.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:15.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:15.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:15.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:15.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:15.409+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:15.429 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:15.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:15.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:15.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:15.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:15.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:16.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:05:16.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:16.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:16.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:16.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:16.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:16.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:16.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:16.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:16.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:17.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:17.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:17.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:17.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:17.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:17.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:17.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:17.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:17.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:17.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:17.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:17.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:20.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:20.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:20.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:20.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:20.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:20.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:20.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:20.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:22.226 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:22.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:22.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:22.273+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:22.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:22.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:22.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:22.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:23.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:23.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:23.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:23.305+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:23.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:23.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:23.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:23.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:23.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:23.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:25.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:25.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:25.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:25.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:25.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:25.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:25.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:25.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:25.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:25.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:25.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:25.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:25.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:25.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:25.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:26.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:26.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:26.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:26.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:26.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:26.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:26.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:26.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:27.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:27.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:27.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:27.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:28.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:28.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:28.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:28.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:28.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:28.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:28.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:28.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:28.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:28.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:28.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:28.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:28.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:28.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:29.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:29.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:29.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:29.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:29.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:29.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:29.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:29.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:29.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:29.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:29.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:29.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:29.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:29.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:29.959 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:29.957+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:30.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:30.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:30.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:30.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:30.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:30.005+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:30.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:30.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:30.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:30.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:30.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:30.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:30.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:30.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:30.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:30.906 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:05:31.626 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:05:31.627 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:05:31.685 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2215,"stamp":"2023-10-14T01:05:31.524590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27548969816,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2075256,"num_read_kb":8055219,"num_write":2034912,"num_write_kb":8140049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80487,"ondisk_log_size":80487,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21107664,"kb_used_data":16276912,"kb_used_omap":1133,"kb_used_meta":4829586,"kb_avail":262007856,"statfs":{"total":289910292480,"available":268296044544,"internally_reserved":0,"allocated":16667557888,"data_stored":20977821833,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1161006,"internal_metadata":4945496274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,8,11,16,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20897792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5131,"num_read_kb":20412,"num_write":5131,"num_write_kb":20524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001977"},"pg_stats":[{"pgid":"2.7","version":"136'272789","reported_seq":535828,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:28.965442+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:05:28.965442+0000","last_peered":"2023-10-14T01:05:28.965442+0000","last_clean":"2023-10-14T01:05:28.965442+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:05:28.965442+0000","last_undegraded":"2023-10-14T01:05:28.965442+0000","last_fullsized":"2023-10-14T01:05:28.965442+0000","mapping_epoch":13,"log_start":"134'262702","ondisk_log_start":"134'262702","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3483660726,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":264729,"num_read_kb":1022015,"num_write":256790,"num_write_kb":1026918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'272926","reported_seq":536655,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:28.958437+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:05:28.958437+0000","last_peered":"2023-10-14T01:05:28.958437+0000","last_clean":"2023-10-14T01:05:28.958437+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:05:28.958437+0000","last_undegraded":"2023-10-14T01:05:28.958437+0000","last_fullsized":"2023-10-14T01:05:28.958437+0000","mapping_epoch":13,"log_start":"134'262909","ondisk_log_start":"134'262909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3477160304,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":267019,"num_read_kb":1021213,"num_write":256933,"num_write_kb":1027359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"136'246929","reported_seq":491393,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:28.951474+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:05:28.951474+0000","last_peered":"2023-10-14T01:05:28.951474+0000","last_clean":"2023-10-14T01:05:28.951474+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:05:28.951474+0000","last_undegraded":"2023-10-14T01:05:28.951474+0000","last_fullsized":"2023-10-14T01:05:28.951474+0000","mapping_epoch":13,"log_start":"134'236905","ondisk_log_start":"134'236905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3326972306,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":244905,"num_read_kb":964280,"num_write":246311,"num_write_kb":985214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"136'241162","reported_seq":482709,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:28.950700+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:05:28.950700+0000","last_peered":"2023-10-14T01:05:28.950700+0000","last_clean":"2023-10-14T01:05:28.950700+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:05:28.950700+0000","last_undegraded":"2023-10-14T01:05:28.950700+0000","last_fullsized":"2023-10-14T01:05:28.950700+0000","mapping_epoch":13,"log_start":"134'231104","ondisk_log_start":"134'231104","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3256373379,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":242025,"num_read_kb":950949,"num_write":240557,"num_write_kb":962150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"136'271772","reported_seq":544529,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:29.743268+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:05:29.743268+0000","last_peered":"2023-10-14T01:05:29.743268+0000","last_clean":"2023-10-14T01:05:29.743268+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:05:29.743268+0000","last_undegraded":"2023-10-14T01:05:29.743268+0000","last_fullsized":"2023-10-14T01:05:29.743268+0000","mapping_epoch":13,"log_start":"134'261707","ondisk_log_start":"134'261707","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":390,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 215s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3661799743,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":273483,"num_read_kb":1068822,"num_write":269996,"num_write_kb":1079837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"136'262381","reported_seq":520826,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:30.859230+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:05:30.859230+0000","last_peered":"2023-10-14T01:05:30.859230+0000","last_clean":"2023-10-14T01:05:30.859230+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:05:30.859230+0000","last_undegraded":"2023-10-14T01:05:30.859230+0000","last_fullsized":"2023-10-14T01:05:30.859230+0000","mapping_epoch":13,"log_start":"134'252307","ondisk_log_start":"134'252307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":560,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 210s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3436511501,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":259999,"num_read_kb":1003848,"num_write":254618,"num_write_kb":1018258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"136'266121","reported_seq":531008,"reported_epoch":136,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:30.843583+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:05:30.843583+0000","last_peered":"2023-10-14T01:05:30.843583+0000","last_clean":"2023-10-14T01:05:30.843583+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:05:30.843583+0000","last_undegraded":"2023-10-14T01:05:30.843583+0000","last_fullsized":"2023-10-14T01:05:30.843583+0000","mapping_epoch":13,"log_start":"134'256110","ondisk_log_start":"134'256110","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":579,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 214s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3483627769,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":266709,"num_read_kb":1024596,"num_write":256699,"num_write_kb":1026522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"136'261231","reported_seq":516467,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:28.922181+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:05:28.922181+0000","last_peered":"2023-10-14T01:05:28.922181+0000","last_clean":"2023-10-14T01:05:28.922181+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:05:28.922181+0000","last_undegraded":"2023-10-14T01:05:28.922181+0000","last_fullsized":"2023-10-14T01:05:28.922181+0000","mapping_epoch":13,"log_start":"134'251207","ondisk_log_start":"134'251207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3422273720,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":256261,"num_read_kb":999387,"num_write":252839,"num_write_kb":1011243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":340,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-14T01:03:36.120433+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:03:36.120433+0000","last_peered":"2023-10-14T01:03:36.120433+0000","last_clean":"2023-10-14T01:03:36.120433+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:03:36.120433+0000","last_undegraded":"2023-10-14T01:03:36.120433+0000","last_fullsized":"2023-10-14T01:03:36.120433+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27548379448,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2075130,"num_read_kb":8055110,"num_write":2034743,"num_write_kb":8137501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16665571328,"data_stored":20977050224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80360,"ondisk_log_size":80360,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739201,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4291836,"kb_used_data":3057076,"kb_used_omap":212,"kb_used_meta":1234539,"kb_avail":90080004,"statfs":{"total":96636764160,"available":92241924096,"internally_reserved":0,"allocated":3130445824,"data_stored":3939657905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1264168180},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:05:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.69899999999999995,"15min":0.84599999999999997},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.14699999999999999},"max":{"1min":5.5419999999999998,"5min":5.6970000000000001,"15min":10.803000000000001},"last":0.185},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.93600000000000005,"15min":0.73599999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":4.306,"5min":10.847,"15min":10.847},"last":0.316}]},{"osd":1,"last update":"Sat Oct 14 01:05:03 2023","interfaces":[{"interface":"back","average":{"1min":2.6619999999999999,"5min":1.5800000000000001,"15min":1.0489999999999999},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.191},"max":{"1min":9.657,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.355,"5min":1.1040000000000001,"15min":1.1279999999999999},"min":{"1min":0.24299999999999999,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":5.2220000000000004,"5min":6.4850000000000003,"15min":16.736999999999998},"last":0.44400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739189,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9721852,"kb_used_data":8137264,"kb_used_omap":565,"kb_used_meta":1584010,"kb_avail":84649988,"statfs":{"total":96636764160,"available":86681587712,"internally_reserved":0,"allocated":8332558336,"data_stored":10488070475,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1622027022},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,4,7,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.65000000000000002,"15min":0.86199999999999999},"min":{"1min":0.29199999999999998,"5min":0.189,"15min":0.16800000000000001},"max":{"1min":1.927,"5min":10.891,"15min":12.369999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.871},"min":{"1min":0.28499999999999998,"5min":0.22,"15min":0.192},"max":{"1min":2.04,"5min":10.856,"15min":12.407},"last":0.374}]},{"osd":2,"last update":"Sat Oct 14 01:04:30 2023","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.92200000000000004,"15min":1.1120000000000001},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.17999999999999999},"max":{"1min":4.6920000000000002,"5min":6.4729999999999999,"15min":14.522},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.80900000000000005,"15min":0.92800000000000005},"min":{"1min":0.216,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":3.988,"5min":7.0599999999999996,"15min":14.537000000000001},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739199,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7093976,"kb_used_data":5082572,"kb_used_omap":355,"kb_used_meta":2011036,"kb_avail":87277864,"statfs":{"total":96636764160,"available":89372532736,"internally_reserved":0,"allocated":5204553728,"data_stored":6550093453,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":364336,"internal_metadata":2059301072},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,3,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:05:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.044,"15min":1.1020000000000001},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":4.9649999999999999,"5min":8.5239999999999991,"15min":12.855},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":1.0980000000000001},"min":{"1min":0.17499999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.774,"5min":6.944,"15min":16.655000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:05:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.569,"15min":1.3129999999999999},"min":{"1min":0.222,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.1710000000000003,"5min":8.9019999999999992,"15min":10.053000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.748,"15min":0.79900000000000004},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.6989999999999998,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5204033536,"data_stored":6549800570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8331771904,"data_stored":10487511352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3129765888,"data_stored":3939738302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:05:31.686 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.687 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.687 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.687 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.688 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.688 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.688 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.689 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:31.689 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:05:32.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:32.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:32.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:32.085+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:32.087 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:32.085+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:34.699 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:34.697+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:05:34.754 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:34.753+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:34.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:34.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:34.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:34.797+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:34.804 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:34.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:34.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:34.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:34.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:34.805+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:36.412 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:36.409+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:36.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:36.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:36.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:36.413+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:37.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:37.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:37.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:37.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:37.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:37.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:37.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:37.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:37.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:37.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:37.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:37.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:39.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:39.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:39.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:39.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:39.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:39.357+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:39.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:39.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:40.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:40.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:40.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:40.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:40.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:40.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:40.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:40.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:40.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:40.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:40.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:40.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:40.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:40.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:41.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:41.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:41.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:41.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:41.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:41.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:42.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:42.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:42.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:42.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:42.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:42.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:42.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:05:42.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:05:42.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:42.489 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:42.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:42.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:42.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:42.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:42.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:42.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:42.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:42.833+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:42.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:42.865+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:42.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:42.865+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:42.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:42.865+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:43.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:43.593+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:43.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:43.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:05:43.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:43.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:43.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:43.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:43.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:43.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:43.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:43.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:43.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:43.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:44.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:44.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:05:44.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:44.637+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:44.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:44.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:46.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:46.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:46.085 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:46.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:48.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:48.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:05:48.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:48.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:48.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:48.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:49.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:49.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:49.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:49.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:49.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:49.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:49.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:49.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:49.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:49.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:50.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:50.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:50.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:50.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:50.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:50.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:50.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:50.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:05:50.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:50.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:50.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:50.280+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:50.332 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:50.328+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:50.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:50.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:50.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:50.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:50.347 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:50.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:51.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:51.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:51.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:51.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:51.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:51.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:51.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:05:51.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:51.087 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:51.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:51.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:51.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:51.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:51.691 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:05:52.486 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:05:52.486 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:05:52.562 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2226,"stamp":"2023-10-14T01:05:51.532302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27583019864,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2083478,"num_read_kb":8087923,"num_write":2043240,"num_write_kb":8173361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80503,"ondisk_log_size":80503,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21199992,"kb_used_data":16343328,"kb_used_omap":1135,"kb_used_meta":4855504,"kb_avail":261915528,"statfs":{"total":289910292480,"available":268201500672,"internally_reserved":0,"allocated":16735567872,"data_stored":21045809122,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1162686,"internal_metadata":4972036674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,8,7,10,12,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20348928,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4857,"num_read_kb":19296,"num_write":4976,"num_write_kb":19904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006281"},"pg_stats":[{"pgid":"2.7","version":"137'273853","reported_seq":537950,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:05:49.396998+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:05:49.396998+0000","last_peered":"2023-10-14T01:05:49.396998+0000","last_clean":"2023-10-14T01:05:49.396998+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:05:49.396998+0000","last_undegraded":"2023-10-14T01:05:49.396998+0000","last_fullsized":"2023-10-14T01:05:49.396998+0000","mapping_epoch":13,"log_start":"134'263802","ondisk_log_start":"134'263802","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3487785398,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":265770,"num_read_kb":1026147,"num_write":257802,"num_write_kb":1030966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'274060","reported_seq":538872,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:05:49.405353+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:05:49.405353+0000","last_peered":"2023-10-14T01:05:49.405353+0000","last_clean":"2023-10-14T01:05:49.405353+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:05:49.405353+0000","last_undegraded":"2023-10-14T01:05:49.405353+0000","last_fullsized":"2023-10-14T01:05:49.405353+0000","mapping_epoch":13,"log_start":"134'264009","ondisk_log_start":"134'264009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3481559408,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":268085,"num_read_kb":1025461,"num_write":258015,"num_write_kb":1031687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'247940","reported_seq":493402,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:05:49.387006+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:05:49.387006+0000","last_peered":"2023-10-14T01:05:49.387006+0000","last_clean":"2023-10-14T01:05:49.387006+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:05:49.387006+0000","last_undegraded":"2023-10-14T01:05:49.387006+0000","last_fullsized":"2023-10-14T01:05:49.387006+0000","mapping_epoch":13,"log_start":"134'237905","ondisk_log_start":"134'237905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3331092882,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":245900,"num_read_kb":968212,"num_write":247322,"num_write_kb":989258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'242170","reported_seq":484707,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:05:49.366386+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:05:49.366386+0000","last_peered":"2023-10-14T01:05:49.366386+0000","last_clean":"2023-10-14T01:05:49.366386+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:05:49.366386+0000","last_undegraded":"2023-10-14T01:05:49.366386+0000","last_fullsized":"2023-10-14T01:05:49.366386+0000","mapping_epoch":13,"log_start":"134'232104","ondisk_log_start":"134'232104","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3260539011,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":243016,"num_read_kb":954897,"num_write":241565,"num_write_kb":966182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"137'272877","reported_seq":546738,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:49.821374+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:05:49.821374+0000","last_peered":"2023-10-14T01:05:49.821374+0000","last_clean":"2023-10-14T01:05:49.821374+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:05:49.821374+0000","last_undegraded":"2023-10-14T01:05:49.821374+0000","last_fullsized":"2023-10-14T01:05:49.821374+0000","mapping_epoch":13,"log_start":"134'262807","ondisk_log_start":"134'262807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":424,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 235s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3666215231,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":274572,"num_read_kb":1073150,"num_write":271081,"num_write_kb":1084177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"137'263463","reported_seq":522958,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:51.177519+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:05:51.177519+0000","last_peered":"2023-10-14T01:05:51.177519+0000","last_clean":"2023-10-14T01:05:51.177519+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:05:51.177519+0000","last_undegraded":"2023-10-14T01:05:51.177519+0000","last_fullsized":"2023-10-14T01:05:51.177519+0000","mapping_epoch":13,"log_start":"134'253407","ondisk_log_start":"134'253407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":620,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 231s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3440660749,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":261030,"num_read_kb":1007960,"num_write":255633,"num_write_kb":1022318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"137'267238","reported_seq":533172,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:05:51.150753+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:05:51.150753+0000","last_peered":"2023-10-14T01:05:51.150753+0000","last_clean":"2023-10-14T01:05:51.150753+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:05:51.150753+0000","last_undegraded":"2023-10-14T01:05:51.150753+0000","last_fullsized":"2023-10-14T01:05:51.150753+0000","mapping_epoch":13,"log_start":"134'257210","ondisk_log_start":"134'257210","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":628,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 234s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3487912185,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":267725,"num_read_kb":1028644,"num_write":257745,"num_write_kb":1030706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"137'262326","reported_seq":518560,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:05:49.386319+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:05:49.386319+0000","last_peered":"2023-10-14T01:05:49.386319+0000","last_clean":"2023-10-14T01:05:49.386319+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:05:49.386319+0000","last_undegraded":"2023-10-14T01:05:49.386319+0000","last_fullsized":"2023-10-14T01:05:49.386319+0000","mapping_epoch":13,"log_start":"134'252307","ondisk_log_start":"134'252307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3426664632,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":257254,"num_read_kb":1003343,"num_write":253908,"num_write_kb":1015519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":342,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:39.145724+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:05:39.145724+0000","last_peered":"2023-10-14T01:05:39.145724+0000","last_clean":"2023-10-14T01:05:39.145724+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:05:39.145724+0000","last_undegraded":"2023-10-14T01:05:39.145724+0000","last_fullsized":"2023-10-14T01:05:39.145724+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27582429496,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2083352,"num_read_kb":8087814,"num_write":2043071,"num_write_kb":8170813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16733683712,"data_stored":21045162608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80376,"ondisk_log_size":80376,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739205,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4360396,"kb_used_data":3069572,"kb_used_omap":212,"kb_used_meta":1290603,"kb_avail":90011444,"statfs":{"total":96636764160,"available":92171718656,"internally_reserved":0,"allocated":3143241728,"data_stored":3952446244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1321577716},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,6,8,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:05:03 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.69899999999999995,"15min":0.84599999999999997},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.14699999999999999},"max":{"1min":5.5419999999999998,"5min":5.6970000000000001,"15min":10.803000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.93600000000000005,"15min":0.73599999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":4.306,"5min":10.847,"15min":10.847},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:05:03 2023","interfaces":[{"interface":"back","average":{"1min":2.6619999999999999,"5min":1.5800000000000001,"15min":1.0489999999999999},"min":{"1min":0.20399999999999999,"5min":0.191,"15min":0.191},"max":{"1min":9.657,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.355,"5min":1.1040000000000001,"15min":1.1279999999999999},"min":{"1min":0.24299999999999999,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":5.2220000000000004,"5min":6.4850000000000003,"15min":16.736999999999998},"last":0.22800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739193,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9815104,"kb_used_data":8170488,"kb_used_omap":565,"kb_used_meta":1644042,"kb_avail":84556736,"statfs":{"total":96636764160,"available":86586097664,"internally_reserved":0,"allocated":8366579712,"data_stored":10522084286,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1683499790},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.65000000000000002,"15min":0.86199999999999999},"min":{"1min":0.29199999999999998,"5min":0.189,"15min":0.16800000000000001},"max":{"1min":1.927,"5min":10.891,"15min":12.369999999999999},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.871},"min":{"1min":0.28499999999999998,"5min":0.22,"15min":0.192},"max":{"1min":2.04,"5min":10.856,"15min":12.407},"last":2.8839999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:05:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.82699999999999996,"15min":1.0740000000000001},"min":{"1min":0.25900000000000001,"5min":0.25,"15min":0.17999999999999999},"max":{"1min":2.8439999999999999,"5min":5.4690000000000003,"15min":14.522},"last":43.871000000000002},{"interface":"front","average":{"1min":1.044,"5min":0.878,"15min":0.93400000000000005},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":7.5780000000000003,"5min":7.5780000000000003,"15min":14.537000000000001},"last":39.412999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739203,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7024492,"kb_used_data":5103268,"kb_used_omap":357,"kb_used_meta":1920858,"kb_avail":87347348,"statfs":{"total":96636764160,"available":89443684352,"internally_reserved":0,"allocated":5225746432,"data_stored":6571278592,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":366016,"internal_metadata":1966959168},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:05:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.044,"15min":1.1020000000000001},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":4.9649999999999999,"5min":8.5239999999999991,"15min":12.855},"last":2.1480000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":1.0980000000000001},"min":{"1min":0.17499999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.774,"5min":6.944,"15min":16.655000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:05:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.569,"15min":1.3129999999999999},"min":{"1min":0.222,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.1710000000000003,"5min":8.9019999999999992,"15min":10.053000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.748,"15min":0.79900000000000004},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.6989999999999998,"5min":8.8249999999999993,"15min":15.451000000000001},"last":2.1909999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5225103360,"data_stored":6570870394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8366051328,"data_stored":10521790776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3142529024,"data_stored":3952501438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:05:52.563 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.564 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.564 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.564 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.565 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.565 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.566 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.566 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:05:52.566 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:05:57.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:57.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:05:57.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:57.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:05:57.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:57.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:57.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:57.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:57.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:57.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:57.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:57.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:05:58.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:58.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:05:58.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:58.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:05:58.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:58.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:05:58.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:58.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:05:58.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:58.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:05:58.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:05:58.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:05:59.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:59.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:05:59.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:59.288+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:05:59.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:59.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:05:59.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:59.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:05:59.296 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:05:59.292+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:05.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:05.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:05.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:05.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:05.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:05.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:07.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:07.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:07.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:07.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:07.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:07.568+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:07.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:07.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:06:07.947 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:07.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:10.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:10.112+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:10.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:10.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:06:10.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:10.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:10.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:10.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:10.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:10.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:10.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:10.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:10.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:10.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:10.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:10.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:11.175 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:11.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:11.185 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:11.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:11.186 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:11.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:12.568 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:06:13.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:13.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:13.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:13.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:13.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:13.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:13.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:13.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:13.422 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:06:13.422 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:06:13.526 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2236,"stamp":"2023-10-14T01:06:11.538669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27617962840,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2092146,"num_read_kb":8122367,"num_write":2051818,"num_write_kb":8207673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80615,"ondisk_log_size":80615,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21522056,"kb_used_data":16411120,"kb_used_omap":1135,"kb_used_meta":5109776,"kb_avail":261593464,"statfs":{"total":289910292480,"available":267871707136,"internally_reserved":0,"allocated":16804986880,"data_stored":21115228130,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1162686,"internal_metadata":5232411202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,2,1,10,13,9,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26365952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6554,"num_read_kb":26044,"num_write":6472,"num_write_kb":25888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004497"},"pg_stats":[{"pgid":"2.7","version":"137'274999","reported_seq":540266,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:09.918687+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:06:09.918687+0000","last_peered":"2023-10-14T01:06:09.918687+0000","last_clean":"2023-10-14T01:06:09.918687+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:06:09.918687+0000","last_undegraded":"2023-10-14T01:06:09.918687+0000","last_fullsized":"2023-10-14T01:06:09.918687+0000","mapping_epoch":13,"log_start":"134'264902","ondisk_log_start":"134'264902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3492295094,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":266924,"num_read_kb":1030727,"num_write":258912,"num_write_kb":1035406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'275222","reported_seq":541130,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:09.919884+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:06:09.919884+0000","last_peered":"2023-10-14T01:06:09.919884+0000","last_clean":"2023-10-14T01:06:09.919884+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:06:09.919884+0000","last_undegraded":"2023-10-14T01:06:09.919884+0000","last_fullsized":"2023-10-14T01:06:09.919884+0000","mapping_epoch":13,"log_start":"134'265209","ondisk_log_start":"134'265209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3486167408,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":269165,"num_read_kb":1029749,"num_write":259143,"num_write_kb":1036199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'248977","reported_seq":495475,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:09.917055+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:06:09.917055+0000","last_peered":"2023-10-14T01:06:09.917055+0000","last_clean":"2023-10-14T01:06:09.917055+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:06:09.917055+0000","last_undegraded":"2023-10-14T01:06:09.917055+0000","last_fullsized":"2023-10-14T01:06:09.917055+0000","mapping_epoch":13,"log_start":"134'238905","ondisk_log_start":"134'238905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3335291282,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":246936,"num_read_kb":972308,"num_write":248359,"num_write_kb":993406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'243186","reported_seq":486759,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:09.896602+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:06:09.896602+0000","last_peered":"2023-10-14T01:06:09.896602+0000","last_clean":"2023-10-14T01:06:09.896602+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:06:09.896602+0000","last_undegraded":"2023-10-14T01:06:09.896602+0000","last_fullsized":"2023-10-14T01:06:09.896602+0000","mapping_epoch":13,"log_start":"134'233104","ondisk_log_start":"134'233104","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3264680067,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":244054,"num_read_kb":959029,"num_write":242581,"num_write_kb":970246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"137'274055","reported_seq":549058,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:10.027417+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:06:10.027417+0000","last_peered":"2023-10-14T01:06:10.027417+0000","last_clean":"2023-10-14T01:06:10.027417+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:06:10.027417+0000","last_undegraded":"2023-10-14T01:06:10.027417+0000","last_fullsized":"2023-10-14T01:06:10.027417+0000","mapping_epoch":13,"log_start":"134'264007","ondisk_log_start":"134'264007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":468,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 255s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3670901055,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":275705,"num_read_kb":1077642,"num_write":272229,"num_write_kb":1088769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"137'264560","reported_seq":525111,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:11.398817+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:06:11.398817+0000","last_peered":"2023-10-14T01:06:11.398817+0000","last_clean":"2023-10-14T01:06:11.398817+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:06:11.398817+0000","last_undegraded":"2023-10-14T01:06:11.398817+0000","last_fullsized":"2023-10-14T01:06:11.398817+0000","mapping_epoch":13,"log_start":"134'254508","ondisk_log_start":"134'254508","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":688,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 251s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3444867341,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":262056,"num_read_kb":1012052,"num_write":256667,"num_write_kb":1026454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"137'268388","reported_seq":535452,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:11.414015+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:06:11.414015+0000","last_peered":"2023-10-14T01:06:11.414015+0000","last_clean":"2023-10-14T01:06:11.414015+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:06:11.414015+0000","last_undegraded":"2023-10-14T01:06:11.414015+0000","last_fullsized":"2023-10-14T01:06:11.414015+0000","mapping_epoch":13,"log_start":"134'258310","ondisk_log_start":"134'258310","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":670,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 255s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3492397305,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":268829,"num_read_kb":1033032,"num_write":258840,"num_write_kb":1035086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"137'263353","reported_seq":520697,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:09.919475+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:06:09.919475+0000","last_peered":"2023-10-14T01:06:09.919475+0000","last_clean":"2023-10-14T01:06:09.919475+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:06:09.919475+0000","last_undegraded":"2023-10-14T01:06:09.919475+0000","last_fullsized":"2023-10-14T01:06:09.919475+0000","mapping_epoch":13,"log_start":"134'253307","ondisk_log_start":"134'253307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3430772920,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":258351,"num_read_kb":1007719,"num_write":254918,"num_write_kb":1019559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":342,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:39.145724+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:05:39.145724+0000","last_peered":"2023-10-14T01:05:39.145724+0000","last_clean":"2023-10-14T01:05:39.145724+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:05:39.145724+0000","last_undegraded":"2023-10-14T01:05:39.145724+0000","last_fullsized":"2023-10-14T01:05:39.145724+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27617372472,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2092020,"num_read_kb":8122258,"num_write":2051649,"num_write_kb":8205125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16803790848,"data_stored":21115269744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80488,"ondisk_log_size":80488,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739209,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4433132,"kb_used_data":3082084,"kb_used_omap":212,"kb_used_meta":1350827,"kb_avail":89938708,"statfs":{"total":96636764160,"available":92097236992,"internally_reserved":0,"allocated":3156054016,"data_stored":3965258532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1383247092},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.71999999999999997,"15min":0.78000000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":3.3279999999999998,"5min":5.6970000000000001,"15min":9.4209999999999994},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.76400000000000001,"15min":0.754},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":7.782,"5min":7.782,"15min":10.847},"last":0.36299999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:06:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.3859999999999999,"15min":1.089},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":5.944,"5min":14.103999999999999,"15min":14.103999999999999},"last":1.129},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.143,"15min":1.1519999999999999},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":7.71,"5min":7.71,"15min":16.736999999999998},"last":0.33400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739197,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9947688,"kb_used_data":8204512,"kb_used_omap":565,"kb_used_meta":1742602,"kb_avail":84424152,"statfs":{"total":96636764160,"available":86450331648,"internally_reserved":0,"allocated":8401420288,"data_stored":10556924862,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1784425230},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,0,5,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:05:17 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.65000000000000002,"15min":0.86199999999999999},"min":{"1min":0.29199999999999998,"5min":0.189,"15min":0.16800000000000001},"max":{"1min":1.927,"5min":10.891,"15min":12.369999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.871},"min":{"1min":0.28499999999999998,"5min":0.22,"15min":0.192},"max":{"1min":2.04,"5min":10.856,"15min":12.407},"last":5.8040000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:05:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.82699999999999996,"15min":1.0740000000000001},"min":{"1min":0.25900000000000001,"5min":0.25,"15min":0.17999999999999999},"max":{"1min":2.8439999999999999,"5min":5.4690000000000003,"15min":14.522},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.044,"5min":0.878,"15min":0.93400000000000005},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":7.5780000000000003,"5min":7.5780000000000003,"15min":14.537000000000001},"last":0.58799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739207,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7141236,"kb_used_data":5124524,"kb_used_omap":357,"kb_used_meta":2016346,"kb_avail":87230604,"statfs":{"total":96636764160,"available":89324138496,"internally_reserved":0,"allocated":5247512576,"data_stored":6593044736,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":366016,"internal_metadata":2064738880},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,1,3,3,2,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:05:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.044,"15min":1.1020000000000001},"min":{"1min":0.193,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":4.9649999999999999,"5min":8.5239999999999991,"15min":12.855},"last":0.182},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.59599999999999997,"15min":1.0980000000000001},"min":{"1min":0.17499999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.774,"5min":6.944,"15min":16.655000000000001},"last":0.216}]},{"osd":2,"last update":"Sat Oct 14 01:05:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6890000000000001,"5min":1.569,"15min":1.3129999999999999},"min":{"1min":0.222,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.1710000000000003,"5min":8.9019999999999992,"15min":10.053000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.748,"15min":0.79900000000000004},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.6989999999999998,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5247090688,"data_stored":6592857722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8401395712,"data_stored":10557135160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3155304448,"data_stored":3965276862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:06:13.527 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.528 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.528 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.528 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.529 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.529 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.530 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.530 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:13.530 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:06:14.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:14.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:14.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:14.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:14.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:14.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:14.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:14.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:14.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:14.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:14.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:14.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:14.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.524+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:14.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:14.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:14.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:14.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:14.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:14.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:14.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:14.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:14.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:14.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:14.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:14.756+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:15.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:06:15.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:06:15.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.008+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:15.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:15.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:15.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:15.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:15.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:15.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:15.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:15.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:15.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:15.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:15.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:15.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:15.932 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:15.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:15.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:15.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:15.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:16.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:16.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:16.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:16.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:18.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:18.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:18.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:18.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:18.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:18.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.276+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:18.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:18.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:18.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:18.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:18.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:18.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:18.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:18.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:18.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:18.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:19.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:06:19.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:19.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:19.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:19.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:19.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:19.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:19.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:19.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:20.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:20.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:20.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:20.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:20.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:20.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:20.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:20.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:20.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:20.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:20.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:20.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:20.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:20.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:21.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:21.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:06:21.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:21.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:21.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:21.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:21.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:21.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:21.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:21.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:21.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:21.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:21.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:21.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:21.871 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:21.868+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:21.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:21.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:21.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:21.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:21.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:21.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:21.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:21.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:25.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:25.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:25.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:25.126 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:25.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:25.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:25.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:25.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:25.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:25.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:27.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:27.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:27.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:27.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:27.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:27.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:27.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:27.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:27.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:27.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:27.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:27.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:29.385 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:06:29.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.388+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:29.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.400+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:29.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.440+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:29.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.440+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:29.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:29.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:29.895 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:29.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:32.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:32.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:32.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:32.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:32.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:32.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:33.531 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:06:34.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:34.216+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:34.242 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:34.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:34.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:34.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:34.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:34.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:34.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:34.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:34.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:34.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:34.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:34.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:34.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:34.384 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:06:34.385 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:06:34.454 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2247,"stamp":"2023-10-14T01:06:33.542306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27651521368,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2100440,"num_read_kb":8155391,"num_write":2060046,"num_write_kb":8240585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80483,"ondisk_log_size":80483,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20433228,"kb_used_data":16478196,"kb_used_omap":1151,"kb_used_meta":3953856,"kb_avail":262682292,"statfs":{"total":289910292480,"available":268986667008,"internally_reserved":0,"allocated":16873672704,"data_stored":21183913954,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1178920,"internal_metadata":4048749272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,13,16,16,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19111936,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4818,"num_read_kb":19196,"num_write":4689,"num_write_kb":18756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001935"},"pg_stats":[{"pgid":"2.7","version":"137'276151","reported_seq":542425,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:30.416293+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:06:30.416293+0000","last_peered":"2023-10-14T01:06:30.416293+0000","last_clean":"2023-10-14T01:06:30.416293+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:06:30.416293+0000","last_undegraded":"2023-10-14T01:06:30.416293+0000","last_fullsized":"2023-10-14T01:06:30.416293+0000","mapping_epoch":13,"log_start":"135'266102","ondisk_log_start":"135'266102","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3496657334,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":267916,"num_read_kb":1034683,"num_write":259982,"num_write_kb":1039686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"137'276339","reported_seq":543322,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:30.418392+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:06:30.418392+0000","last_peered":"2023-10-14T01:06:30.418392+0000","last_clean":"2023-10-14T01:06:30.418392+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:06:30.418392+0000","last_undegraded":"2023-10-14T01:06:30.418392+0000","last_fullsized":"2023-10-14T01:06:30.418392+0000","mapping_epoch":13,"log_start":"135'266309","ondisk_log_start":"135'266309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3490394480,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":270224,"num_read_kb":1033949,"num_write":260179,"num_write_kb":1040343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"137'249950","reported_seq":497446,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:30.424674+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:06:30.424674+0000","last_peered":"2023-10-14T01:06:30.424674+0000","last_clean":"2023-10-14T01:06:30.424674+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:06:30.424674+0000","last_undegraded":"2023-10-14T01:06:30.424674+0000","last_fullsized":"2023-10-14T01:06:30.424674+0000","mapping_epoch":13,"log_start":"134'239905","ondisk_log_start":"134'239905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3339280786,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":247935,"num_read_kb":976296,"num_write":249332,"num_write_kb":997298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'244174","reported_seq":488747,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:30.418709+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:06:30.418709+0000","last_peered":"2023-10-14T01:06:30.418709+0000","last_clean":"2023-10-14T01:06:30.418709+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:06:30.418709+0000","last_undegraded":"2023-10-14T01:06:30.418709+0000","last_fullsized":"2023-10-14T01:06:30.418709+0000","mapping_epoch":13,"log_start":"134'234104","ondisk_log_start":"134'234104","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3268690051,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":245053,"num_read_kb":963009,"num_write":243569,"num_write_kb":974198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"137'275219","reported_seq":551300,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:30.205319+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:06:30.205319+0000","last_peered":"2023-10-14T01:06:30.205319+0000","last_clean":"2023-10-14T01:06:30.205319+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:06:30.205319+0000","last_undegraded":"2023-10-14T01:06:30.205319+0000","last_fullsized":"2023-10-14T01:06:30.205319+0000","mapping_epoch":13,"log_start":"135'265207","ondisk_log_start":"135'265207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":524,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 275s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3675521343,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":276771,"num_read_kb":1081890,"num_write":273363,"num_write_kb":1093305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"137'265650","reported_seq":527310,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:31.728457+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:06:31.728457+0000","last_peered":"2023-10-14T01:06:31.728457+0000","last_clean":"2023-10-14T01:06:31.728457+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:06:31.728457+0000","last_undegraded":"2023-10-14T01:06:31.728457+0000","last_fullsized":"2023-10-14T01:06:31.728457+0000","mapping_epoch":13,"log_start":"135'255608","ondisk_log_start":"135'255608","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":745,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 271s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3449082125,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":263139,"num_read_kb":1016376,"num_write":257702,"num_write_kb":1030594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"137'269445","reported_seq":537585,"reported_epoch":137,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:31.730989+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:06:31.730989+0000","last_peered":"2023-10-14T01:06:31.730989+0000","last_clean":"2023-10-14T01:06:31.730989+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:06:31.730989+0000","last_undegraded":"2023-10-14T01:06:31.730989+0000","last_fullsized":"2023-10-14T01:06:31.730989+0000","mapping_epoch":13,"log_start":"135'259410","ondisk_log_start":"135'259410","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":710,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 275s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3496505593,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":269876,"num_read_kb":1037192,"num_write":259846,"num_write_kb":1039110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"137'264380","reported_seq":522781,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:06:30.389848+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:06:30.389848+0000","last_peered":"2023-10-14T01:06:30.389848+0000","last_clean":"2023-10-14T01:06:30.389848+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:06:30.389848+0000","last_undegraded":"2023-10-14T01:06:30.389848+0000","last_fullsized":"2023-10-14T01:06:30.389848+0000","mapping_epoch":13,"log_start":"135'254307","ondisk_log_start":"135'254307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3434799288,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":259400,"num_read_kb":1011887,"num_write":255904,"num_write_kb":1023503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":342,"reported_epoch":136,"state":"active+clean","last_fresh":"2023-10-14T01:05:39.145724+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:05:39.145724+0000","last_peered":"2023-10-14T01:05:39.145724+0000","last_clean":"2023-10-14T01:05:39.145724+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:05:39.145724+0000","last_undegraded":"2023-10-14T01:05:39.145724+0000","last_fullsized":"2023-10-14T01:05:39.145724+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27650931000,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2100314,"num_read_kb":8155282,"num_write":2059877,"num_write_kb":8238037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16870998016,"data_stored":21182476912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80356,"ondisk_log_size":80356,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739213,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4494224,"kb_used_data":3094600,"kb_used_omap":212,"kb_used_meta":1399403,"kb_avail":89877616,"statfs":{"total":96636764160,"available":92034678784,"internally_reserved":0,"allocated":3168870400,"data_stored":3978074916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1432988916},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.71999999999999997,"15min":0.78000000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":3.3279999999999998,"5min":5.6970000000000001,"15min":9.4209999999999994},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.76400000000000001,"15min":0.754},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":7.782,"5min":7.782,"15min":10.847},"last":0.5}]},{"osd":1,"last update":"Sat Oct 14 01:06:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.3859999999999999,"15min":1.089},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":5.944,"5min":14.103999999999999,"15min":14.103999999999999},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.143,"15min":1.1519999999999999},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":7.71,"5min":7.71,"15min":16.736999999999998},"last":0.54400000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739201,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9981904,"kb_used_data":8238472,"kb_used_omap":565,"kb_used_meta":1742858,"kb_avail":84389936,"statfs":{"total":96636764160,"available":86415294464,"internally_reserved":0,"allocated":8436195328,"data_stored":10591699902,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1784687374},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,9,11,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:06:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.53800000000000003,"15min":0.81599999999999995},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.47199999999999998,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.624,"15min":0.85099999999999998},"min":{"1min":0.28000000000000003,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":5.8040000000000003,"5min":5.8040000000000003,"15min":12.407},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:05:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.82699999999999996,"15min":1.0740000000000001},"min":{"1min":0.25900000000000001,"5min":0.25,"15min":0.17999999999999999},"max":{"1min":2.8439999999999999,"5min":5.4690000000000003,"15min":14.522},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.044,"5min":0.878,"15min":0.93400000000000005},"min":{"1min":0.223,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":7.5780000000000003,"5min":7.5780000000000003,"15min":14.537000000000001},"last":0.48599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739211,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5957100,"kb_used_data":5145124,"kb_used_omap":373,"kb_used_meta":811594,"kb_avail":88414740,"statfs":{"total":96636764160,"available":90536693760,"internally_reserved":0,"allocated":5268606976,"data_stored":6614139136,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":831072982},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:06:12 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.004,"15min":1.087},"min":{"1min":0.182,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":6.7290000000000001,"5min":6.7290000000000001,"15min":12.855},"last":0.432},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.56699999999999995,"15min":1.006},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.1759999999999999,"5min":6.944,"15min":16.655000000000001},"last":0.20899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:06:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.5629999999999999,"15min":1.3380000000000001},"min":{"1min":0.214,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0119999999999996,"5min":8.9019999999999992,"15min":10.053000000000001},"last":4.0099999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.83699999999999997,"15min":0.78400000000000003},"min":{"1min":0.24299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.2329999999999997,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5268099072,"data_stored":6613866106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8435163136,"data_stored":10590902584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3167735808,"data_stored":3977708222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:06:34.455 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.456 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.456 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.457 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.457 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.457 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.458 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.458 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:06:34.459 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:06:34.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:34.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:34.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:34.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:34.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:34.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:34.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:34.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:34.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:34.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:35.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:35.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:35.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:35.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:35.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:35.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:35.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:35.128+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:35.472 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:06:35.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:35.515+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:35.530 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-14T01:06:35.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:35.571+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:35.575 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:35.571+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:35.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:35.571+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:36.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:36.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:36.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:36.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:36.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:36.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:36.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:36.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:36.414 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-14T01:06:36.452 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:06:37.301 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:06:37.344 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-14T01:06:37.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:37.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:06:37.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:37.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:37.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:37.827+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:37.836 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:37.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:37.836 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:37.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:37.836 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:37.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:38.253 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-14T01:06:38.324 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-10-14T01:06:39.221 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-10-14T01:06:39.279 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-10-14T01:06:39.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:39.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:39.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:39.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:39.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:39.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:39.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:39.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:39.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:39.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:39.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:39.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:40.012 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-10-14T01:06:40.054 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-14T01:06:40.917 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-14T01:06:40.973 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-14T01:06:41.746 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-14T01:06:41.828 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:06:42.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:42.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:42.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:42.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:42.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:42.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:42.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:42.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:42.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:42.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:42.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:43.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:43.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:06:43.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:43.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:43.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:43.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:43.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:43.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:43.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:43.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:43.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:06:43.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:43.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.655+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:43.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:43.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:43.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:43.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.955+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:43.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:43.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:44.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:44.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:44.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:44.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:44.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:44.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:44.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:44.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:45.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:45.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:45.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:45.551+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:45.566 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:45.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:45.570 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:45.567+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:45.571 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:45.567+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:46.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:46.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:46.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:46.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:46.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:46.627+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:06:46.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:46.627+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:06:46.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:46.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:46.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:46.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:46.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:46.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:46.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:46.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:46.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:46.699+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:46.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:46.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:46.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:46.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:46.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:46.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:46.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:46.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:48.096 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:48.095+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:06:48.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:48.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:06:48.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:48.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:48.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:48.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:48.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:48.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:48.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:48.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:48.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:48.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:48.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:48.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:48.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:48.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:49.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:49.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:49.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:49.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:49.219 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:49.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:49.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:49.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:50.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:06:50.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:50.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:50.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:50.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:50.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:50.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.607+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:50.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:50.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:50.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:50.687+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:50.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:50.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:50.725 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:50.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:50.725 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:50.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:51.804 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:51.799+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:51.819 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:51.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:51.819 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:51.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:51.820 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:51.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:53.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:53.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:53.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:53.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:53.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:53.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:53.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:53.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:53.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:53.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:55.470 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:55.463+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:06:55.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:55.487+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:06:55.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:55.487+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:06:55.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:06:55.487+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:06:56.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:56.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:56.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:56.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:56.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:06:56.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:56.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:56.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:56.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:56.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:56.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:56.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:57.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:57.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:57.665 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:57.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:57.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:57.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:57.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:57.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:57.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:57.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:06:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:59.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:06:59.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:59.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:06:59.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:59.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:06:59.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:59.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:06:59.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:59.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:06:59.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:06:59.551+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:00.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:00.695+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:07:00.697 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:00.695+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:00.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:00.699+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:00.704 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:00.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:00.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:00.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:00.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:00.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:00.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:00.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:00.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:00.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:00.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:00.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:00.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:00.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:00.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:00.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:00.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:00.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:00.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:00.923+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:00.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:00.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:01.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:01.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:01.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:01.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:01.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:01.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:01.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:01.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:01.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:01.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:01.830 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:07:02.642 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:07:02.644 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:07:02.696 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2262,"stamp":"2023-10-14T01:07:01.551775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27700239192,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2112310,"num_read_kb":8202615,"num_write":2072004,"num_write_kb":8288417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80567,"ondisk_log_size":80567,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20755832,"kb_used_data":16574240,"kb_used_omap":1151,"kb_used_meta":4180416,"kb_avail":262359688,"statfs":{"total":289910292480,"available":268656320512,"internally_reserved":0,"allocated":16972021760,"data_stored":21282240186,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1178920,"internal_metadata":4280746712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,6,20,16,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17547264,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4289,"num_read_kb":17052,"num_write":4294,"num_write_kb":17176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002637"},"pg_stats":[{"pgid":"2.7","version":"138'277904","reported_seq":545782,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:01.002599+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:07:01.002599+0000","last_peered":"2023-10-14T01:07:01.002599+0000","last_clean":"2023-10-14T01:07:01.002599+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:07:01.002599+0000","last_undegraded":"2023-10-14T01:07:01.002599+0000","last_fullsized":"2023-10-14T01:07:01.002599+0000","mapping_epoch":13,"log_start":"135'267902","ondisk_log_start":"135'267902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3503358390,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":269499,"num_read_kb":1040987,"num_write":261627,"num_write_kb":1046266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'278047","reported_seq":546606,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:01.000418+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:07:01.000418+0000","last_peered":"2023-10-14T01:07:01.000418+0000","last_clean":"2023-10-14T01:07:01.000418+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:07:01.000418+0000","last_undegraded":"2023-10-14T01:07:01.000418+0000","last_fullsized":"2023-10-14T01:07:01.000418+0000","mapping_epoch":13,"log_start":"135'268009","ondisk_log_start":"135'268009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3496919408,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":271778,"num_read_kb":1040121,"num_write":261779,"num_write_kb":1046743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'251447","reported_seq":500419,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:00.951123+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:07:00.951123+0000","last_peered":"2023-10-14T01:07:00.951123+0000","last_clean":"2023-10-14T01:07:00.951123+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:07:00.951123+0000","last_undegraded":"2023-10-14T01:07:00.951123+0000","last_fullsized":"2023-10-14T01:07:00.951123+0000","mapping_epoch":13,"log_start":"135'241405","ondisk_log_start":"135'241405","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3345387922,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":249410,"num_read_kb":982172,"num_write":250829,"num_write_kb":1003286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'245603","reported_seq":491628,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:00.984374+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:07:00.984374+0000","last_peered":"2023-10-14T01:07:00.984374+0000","last_clean":"2023-10-14T01:07:00.984374+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:07:00.984374+0000","last_undegraded":"2023-10-14T01:07:00.984374+0000","last_fullsized":"2023-10-14T01:07:00.984374+0000","mapping_epoch":13,"log_start":"135'235504","ondisk_log_start":"135'235504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3274502275,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":246502,"num_read_kb":968765,"num_write":244998,"num_write_kb":979914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"138'276892","reported_seq":554647,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:07:00.533134+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:07:00.533134+0000","last_peered":"2023-10-14T01:07:00.533134+0000","last_clean":"2023-10-14T01:07:00.533134+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:07:00.533134+0000","last_undegraded":"2023-10-14T01:07:00.533134+0000","last_fullsized":"2023-10-14T01:07:00.533134+0000","mapping_epoch":13,"log_start":"135'266807","ondisk_log_start":"135'266807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":597,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 305s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3682136383,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":278425,"num_read_kb":1088478,"num_write":274993,"num_write_kb":1099825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"138'267059","reported_seq":530046,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:57.075107+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:06:57.075107+0000","last_peered":"2023-10-14T01:06:57.075107+0000","last_clean":"2023-10-14T01:06:57.075107+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:06:57.075107+0000","last_undegraded":"2023-10-14T01:06:57.075107+0000","last_fullsized":"2023-10-14T01:06:57.075107+0000","mapping_epoch":13,"log_start":"135'257008","ondisk_log_start":"135'257008","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":816,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 297s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3454402829,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":264434,"num_read_kb":1021540,"num_write":259011,"num_write_kb":1035830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"138'270865","reported_seq":540386,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:06:57.062860+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:06:57.062860+0000","last_peered":"2023-10-14T01:06:57.062860+0000","last_clean":"2023-10-14T01:06:57.062860+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:06:57.062860+0000","last_undegraded":"2023-10-14T01:06:57.062860+0000","last_fullsized":"2023-10-14T01:06:57.062860+0000","mapping_epoch":13,"log_start":"135'260810","ondisk_log_start":"135'260810","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":768,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 300s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3501867257,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":271224,"num_read_kb":1042536,"num_write":261153,"num_write_kb":1044338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"138'265975","reported_seq":525898,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:00.928788+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:07:00.928788+0000","last_peered":"2023-10-14T01:07:00.928788+0000","last_clean":"2023-10-14T01:07:00.928788+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:07:00.928788+0000","last_undegraded":"2023-10-14T01:07:00.928788+0000","last_fullsized":"2023-10-14T01:07:00.928788+0000","mapping_epoch":13,"log_start":"135'255907","ondisk_log_start":"135'255907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3441074360,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":260912,"num_read_kb":1017907,"num_write":257445,"num_write_kb":1029667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":343,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:05:39.427521+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:05:39.427521+0000","last_peered":"2023-10-14T01:05:39.427521+0000","last_clean":"2023-10-14T01:05:39.427521+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:05:39.427521+0000","last_undegraded":"2023-10-14T01:05:39.427521+0000","last_fullsized":"2023-10-14T01:05:39.427521+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27699648824,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2112184,"num_read_kb":8202506,"num_write":2071835,"num_write_kb":8285869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16970047488,"data_stored":21281526384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80440,"ondisk_log_size":80440,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739218,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4559236,"kb_used_data":3109628,"kb_used_omap":212,"kb_used_meta":1449387,"kb_avail":89812604,"statfs":{"total":96636764160,"available":91968106496,"internally_reserved":0,"allocated":3184259072,"data_stored":3993455980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1484172532},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,3,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.71999999999999997,"15min":0.78000000000000003},"min":{"1min":0.185,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":3.3279999999999998,"5min":5.6970000000000001,"15min":9.4209999999999994},"last":0.31},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.76400000000000001,"15min":0.754},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":7.782,"5min":7.782,"15min":10.847},"last":0.27600000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:06:04 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.3859999999999999,"15min":1.089},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.191},"max":{"1min":5.944,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.248},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.143,"15min":1.1519999999999999},"min":{"1min":0.22800000000000001,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":7.71,"5min":7.71,"15min":16.736999999999998},"last":1.3520000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739207,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10204788,"kb_used_data":8287724,"kb_used_omap":565,"kb_used_meta":1916490,"kb_avail":84167052,"statfs":{"total":96636764160,"available":86187061248,"internally_reserved":0,"allocated":8486629376,"data_stored":10642126342,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":578802,"internal_metadata":1962486542},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:06:18 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.53800000000000003,"15min":0.81599999999999995},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.47199999999999998,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.624,"15min":0.85099999999999998},"min":{"1min":0.28000000000000003,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":5.8040000000000003,"5min":5.8040000000000003,"15min":12.407},"last":0.38900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:06:34 2023","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":1.1950000000000001,"15min":1.1619999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":43.871000000000002,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":2.7650000000000001,"5min":1.1990000000000001,"15min":1.0660000000000001},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":39.412999999999997,"5min":39.412999999999997,"15min":39.412999999999997},"last":4.6879999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739217,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5991808,"kb_used_data":5176888,"kb_used_omap":373,"kb_used_meta":814538,"kb_avail":88380032,"statfs":{"total":96636764160,"available":90501152768,"internally_reserved":0,"allocated":5301133312,"data_stored":6646657864,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":834087638},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,7,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:06:12 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.004,"15min":1.087},"min":{"1min":0.182,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":6.7290000000000001,"5min":6.7290000000000001,"15min":12.855},"last":0.214},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.56699999999999995,"15min":1.006},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.1759999999999999,"5min":6.944,"15min":16.655000000000001},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:06:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.5629999999999999,"15min":1.3380000000000001},"min":{"1min":0.214,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":7.0119999999999996,"5min":8.9019999999999992,"15min":10.053000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.83699999999999997,"15min":0.78400000000000003},"min":{"1min":0.24299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.2329999999999997,"5min":8.8249999999999993,"15min":15.451000000000001},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5299884032,"data_stored":6645651066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8486502400,"data_stored":10642241848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3183661056,"data_stored":3993633470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:07:02.697 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.698 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.698 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.699 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.699 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.699 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.700 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.700 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:02.700 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:07:04.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:04.735+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:04.756 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:04.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:04.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:04.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:04.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:04.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:04.757 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:04.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:05.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:05.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:05.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:05.059+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:05.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:05.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:05.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:05.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:05.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:05.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:05.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:05.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:05.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:05.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:05.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:05.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:06.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:06.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:06.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:06.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:06.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:06.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:06.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:06.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:06.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:06.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:06.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:06.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:06.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:07.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:07.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:07.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:07.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:07.643 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:07.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:07.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:07.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:07.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:07.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:07.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:07.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:07.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:07.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:07.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:07.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:07.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:07.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:07.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:07.947+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:08.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:08.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:08.033 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:08.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:08.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:08.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:08.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:08.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:10.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:07:10.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:10.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:10.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:10.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:10.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:10.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.371+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:10.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:10.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:10.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:10.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:07:10.517 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:10.515+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:10.520 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:10.519+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:10.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:10.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:10.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:10.539+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:10.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:10.539+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:10.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:10.539+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:12.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:12.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:12.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:12.159+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:12.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:12.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:12.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:12.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:12.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:12.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:12.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:12.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:12.993 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:12.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:13.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:13.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:13.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:13.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:14.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:14.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:14.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:14.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:13.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:15.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:07:15.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:07:15.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:07:15.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:15.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:15.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:15.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.687+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:15.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:15.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:15.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:15.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:16.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:16.183+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:07:16.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:16.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:16.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:16.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:16.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:16.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:16.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:16.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:16.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:16.975+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:16.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:16.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:17.009 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:17.003+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:17.009 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:17.007+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:17.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:17.007+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:17.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:17.007+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:19.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:19.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:19.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:19.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:19.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:19.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:19.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:19.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:19.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:19.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:19.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:19.079+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:21.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:21.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:21.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:21.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:21.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:21.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:21.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:21.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:21.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:21.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:22.701 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:07:23.670 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:07:23.671 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:07:23.709 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2273,"stamp":"2023-10-14T01:07:23.555976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27737398104,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2121364,"num_read_kb":8238639,"num_write":2081102,"num_write_kb":8324809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80492,"ondisk_log_size":80492,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20555100,"kb_used_data":16646720,"kb_used_omap":1219,"kb_used_meta":3907132,"kb_avail":262560420,"statfs":{"total":289910292480,"available":268861870080,"internally_reserved":0,"allocated":17046241280,"data_stored":21356459706,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1249123,"internal_metadata":4000903325},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,7,8,18,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20000768,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4878,"num_read_kb":19416,"num_write":4893,"num_write_kb":19572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002520"},"pg_stats":[{"pgid":"2.7","version":"138'279080","reported_seq":548074,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:21.462983+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:07:21.462983+0000","last_peered":"2023-10-14T01:07:21.462983+0000","last_clean":"2023-10-14T01:07:21.462983+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:07:21.462983+0000","last_undegraded":"2023-10-14T01:07:21.462983+0000","last_fullsized":"2023-10-14T01:07:21.462983+0000","mapping_epoch":13,"log_start":"135'269002","ondisk_log_start":"135'269002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3507950006,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":270598,"num_read_kb":1045367,"num_write":262751,"num_write_kb":1050762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'279125","reported_seq":548733,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:21.382420+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:07:21.382420+0000","last_peered":"2023-10-14T01:07:21.382420+0000","last_clean":"2023-10-14T01:07:21.382420+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:07:21.382420+0000","last_undegraded":"2023-10-14T01:07:21.382420+0000","last_fullsized":"2023-10-14T01:07:21.382420+0000","mapping_epoch":13,"log_start":"135'269109","ondisk_log_start":"135'269109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3501080944,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":272809,"num_read_kb":1044229,"num_write":262805,"num_write_kb":1050847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'252459","reported_seq":502470,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:21.399748+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:07:21.399748+0000","last_peered":"2023-10-14T01:07:21.399748+0000","last_clean":"2023-10-14T01:07:21.399748+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:07:21.399748+0000","last_undegraded":"2023-10-14T01:07:21.399748+0000","last_fullsized":"2023-10-14T01:07:21.399748+0000","mapping_epoch":13,"log_start":"135'242405","ondisk_log_start":"135'242405","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3349512594,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":250446,"num_read_kb":986308,"num_write":251841,"num_write_kb":1007334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'246636","reported_seq":493656,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:21.422139+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:07:21.422139+0000","last_peered":"2023-10-14T01:07:21.422139+0000","last_clean":"2023-10-14T01:07:21.422139+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:07:21.422139+0000","last_undegraded":"2023-10-14T01:07:21.422139+0000","last_fullsized":"2023-10-14T01:07:21.422139+0000","mapping_epoch":13,"log_start":"135'236604","ondisk_log_start":"135'236604","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3278733443,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":247497,"num_read_kb":972729,"num_write":246031,"num_write_kb":984046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"138'278074","reported_seq":556994,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:07:20.678114+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:07:20.678114+0000","last_peered":"2023-10-14T01:07:20.678114+0000","last_clean":"2023-10-14T01:07:20.678114+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:07:20.678114+0000","last_undegraded":"2023-10-14T01:07:20.678114+0000","last_fullsized":"2023-10-14T01:07:20.678114+0000","mapping_epoch":13,"log_start":"135'268007","ondisk_log_start":"135'268007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":634,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 326s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3686863167,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":279578,"num_read_kb":1093054,"num_write":276150,"num_write_kb":1104453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"138'268480","reported_seq":532782,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:07:22.446842+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:07:22.446842+0000","last_peered":"2023-10-14T01:07:22.446842+0000","last_clean":"2023-10-14T01:07:22.446842+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:07:22.446842+0000","last_undegraded":"2023-10-14T01:07:22.446842+0000","last_fullsized":"2023-10-14T01:07:22.446842+0000","mapping_epoch":13,"log_start":"135'258409","ondisk_log_start":"135'258409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":881,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 322s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3459858701,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":265714,"num_read_kb":1026624,"num_write":260348,"num_write_kb":1041178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"138'272317","reported_seq":543281,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:07:22.385786+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:07:22.385786+0000","last_peered":"2023-10-14T01:07:22.385786+0000","last_clean":"2023-10-14T01:07:22.385786+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:07:22.385786+0000","last_undegraded":"2023-10-14T01:07:22.385786+0000","last_fullsized":"2023-10-14T01:07:22.385786+0000","mapping_epoch":13,"log_start":"135'262310","ondisk_log_start":"135'262310","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":817,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 326s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3507446009,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":272631,"num_read_kb":1048120,"num_write":262516,"num_write_kb":1049790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"138'267047","reported_seq":528027,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:21.392283+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:07:21.392283+0000","last_peered":"2023-10-14T01:07:21.392283+0000","last_clean":"2023-10-14T01:07:21.392283+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:07:21.392283+0000","last_undegraded":"2023-10-14T01:07:21.392283+0000","last_fullsized":"2023-10-14T01:07:21.392283+0000","mapping_epoch":13,"log_start":"135'257007","ondisk_log_start":"135'257007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3445362872,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":261965,"num_read_kb":1022099,"num_write":258491,"num_write_kb":1033851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":343,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-10-14T01:05:39.427521+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:05:39.427521+0000","last_peered":"2023-10-14T01:05:39.427521+0000","last_clean":"2023-10-14T01:05:39.427521+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:05:39.427521+0000","last_undegraded":"2023-10-14T01:05:39.427521+0000","last_fullsized":"2023-10-14T01:05:39.427521+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27736807736,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2121238,"num_read_kb":8238530,"num_write":2080933,"num_write_kb":8322261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17043283968,"data_stored":21354762864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80365,"ondisk_log_size":80365,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4629376,"kb_used_data":3125624,"kb_used_omap":212,"kb_used_meta":1503531,"kb_avail":89742464,"statfs":{"total":96636764160,"available":91896283136,"internally_reserved":0,"allocated":3200638976,"data_stored":4009835884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1539615988},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:07:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":0.85999999999999999,"15min":0.83099999999999996},"min":{"1min":0.19400000000000001,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":7.492,"5min":7.492,"15min":9.4209999999999994},"last":5.2880000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.77800000000000002},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":7.415,"5min":7.782,"15min":10.847},"last":5.3739999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:07:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.405,"15min":1.1359999999999999},"min":{"1min":0.246,"5min":0.191,"15min":0.191},"max":{"1min":5.7919999999999998,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":1.121,"15min":1.1439999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":1.3520000000000001,"5min":7.71,"15min":16.736999999999998},"last":5.335}]}]},{"osd":1,"up_from":8,"seq":34359739211,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9970888,"kb_used_data":8323008,"kb_used_omap":633,"kb_used_meta":1647238,"kb_avail":84400952,"statfs":{"total":96636764160,"available":86426574848,"internally_reserved":0,"allocated":8522760192,"data_stored":10678257158,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":649005,"internal_metadata":1686771923},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,7,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:07:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.53000000000000003,"15min":0.80000000000000004},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":2.9300000000000002,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.62,"15min":0.83499999999999996},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":3.1080000000000001,"5min":5.8040000000000003,"15min":12.407},"last":0.39000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:06:34 2023","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":1.1950000000000001,"15min":1.1619999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":43.871000000000002,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7650000000000001,"5min":1.1990000000000001,"15min":1.0660000000000001},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":39.412999999999997,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.26700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739221,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5954836,"kb_used_data":5198088,"kb_used_omap":373,"kb_used_meta":756362,"kb_avail":88417004,"statfs":{"total":96636764160,"available":90539012096,"internally_reserved":0,"allocated":5322842112,"data_stored":6668366664,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":774515414},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:07:12 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.84099999999999997,"15min":0.995},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":0.60399999999999998,"5min":6.7290000000000001,"15min":12.855},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.52800000000000002,"15min":0.92500000000000004},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.497,"5min":6.944,"15min":16.655000000000001},"last":0.25700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:07:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.496,"15min":1.379},"min":{"1min":0.23899999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":10.539999999999999,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.68899999999999995,"15min":0.77100000000000002},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.1230000000000002,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5321519104,"data_stored":6667286138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8521732096,"data_stored":10677471544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3200032768,"data_stored":4010005182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:07:23.711 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.711 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.712 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.712 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.712 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.713 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.713 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.714 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:23.714 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:07:23.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:23.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:23.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:23.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:23.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:23.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:23.778 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:23.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:23.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:23.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:23.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:23.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:25.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:25.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:25.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:25.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:25.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:25.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:25.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:25.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:25.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:25.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:26.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:26.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:26.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:26.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:26.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:26.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:26.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:26.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:26.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:26.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:27.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:27.082+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:27.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:27.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:27.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:27.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:27.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:27.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:27.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:27.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:27.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:27.630+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:27.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:27.646+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:27.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:27.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:27.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:27.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:27.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:27.674+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:28.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:07:28.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:07:28.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:28.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:28.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:28.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:28.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:28.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:28.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:29.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:29.070+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:29.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:29.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:29.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:29.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:29.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:29.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:29.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:29.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:29.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:29.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:29.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:29.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:30.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:30.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:30.436 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:30.430+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:30.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:30.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:30.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:30.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:30.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:30.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:30.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:30.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:30.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:30.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:30.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:30.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:30.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:30.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:30.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:30.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:30.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:30.666+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:31.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:31.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:31.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:31.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:31.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:31.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:31.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:31.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:31.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:31.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:31.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:31.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:32.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:32.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:32.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:32.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:32.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:07:32.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:32.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:32.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:33.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:33.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:32.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:34.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:34.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:34.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:34.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:34.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:34.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:34.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:34.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:34.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:34.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:34.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:34.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:35.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:35.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:35.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:35.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:35.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:35.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:35.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:35.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:35.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:35.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:35.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:35.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:35.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:35.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:35.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:35.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:35.835 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:35.830+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:35.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:35.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:35.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:35.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:35.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:35.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:35.857 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:35.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:36.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:36.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:36.196 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:36.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:36.196 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:36.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:36.197 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:36.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:36.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:36.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:36.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:36.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:36.409 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:36.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:36.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:36.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:36.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:36.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:36.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:36.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:42.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:42.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:42.281 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:42.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:42.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:42.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:42.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:42.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:42.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:42.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:42.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:42.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:43.715 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:07:43.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:43.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:07:43.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:43.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:43.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:43.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:43.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:43.822+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:43.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:43.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:43.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:43.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:43.829 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:43.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:44.676 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:07:44.677 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:07:44.718 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2283,"stamp":"2023-10-14T01:07:43.559313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27772123992,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2129923,"num_read_kb":8272735,"num_write":2089613,"num_write_kb":8358853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80483,"ondisk_log_size":80483,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20833268,"kb_used_data":16713880,"kb_used_omap":1219,"kb_used_meta":4118140,"kb_avail":262282252,"statfs":{"total":289910292480,"available":268577026048,"internally_reserved":0,"allocated":17115013120,"data_stored":21425231546,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1249123,"internal_metadata":4216975517},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,4,12,14,19,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24907776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6125,"num_read_kb":24424,"num_write":6105,"num_write_kb":24420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002118"},"pg_stats":[{"pgid":"2.7","version":"138'280228","reported_seq":550311,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.841244+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:07:41.841244+0000","last_peered":"2023-10-14T01:07:41.841244+0000","last_clean":"2023-10-14T01:07:41.841244+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:07:41.841244+0000","last_undegraded":"2023-10-14T01:07:41.841244+0000","last_fullsized":"2023-10-14T01:07:41.841244+0000","mapping_epoch":13,"log_start":"136'270202","ondisk_log_start":"136'270202","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3512377782,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":271669,"num_read_kb":1049635,"num_write":263838,"num_write_kb":1055110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'280277","reported_seq":551032,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.840985+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:07:41.840985+0000","last_peered":"2023-10-14T01:07:41.840985+0000","last_clean":"2023-10-14T01:07:41.840985+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:07:41.840985+0000","last_undegraded":"2023-10-14T01:07:41.840985+0000","last_fullsized":"2023-10-14T01:07:41.840985+0000","mapping_epoch":13,"log_start":"136'270209","ondisk_log_start":"136'270209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3505533296,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":273940,"num_read_kb":1048737,"num_write":263897,"num_write_kb":1055215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'253486","reported_seq":504511,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.814865+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:07:41.814865+0000","last_peered":"2023-10-14T01:07:41.814865+0000","last_clean":"2023-10-14T01:07:41.814865+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:07:41.814865+0000","last_undegraded":"2023-10-14T01:07:41.814865+0000","last_fullsized":"2023-10-14T01:07:41.814865+0000","mapping_epoch":13,"log_start":"135'243405","ondisk_log_start":"135'243405","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3353706898,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":251460,"num_read_kb":990344,"num_write":252868,"num_write_kb":1011442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'247644","reported_seq":495707,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.804794+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:07:41.804794+0000","last_peered":"2023-10-14T01:07:41.804794+0000","last_clean":"2023-10-14T01:07:41.804794+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:07:41.804794+0000","last_undegraded":"2023-10-14T01:07:41.804794+0000","last_fullsized":"2023-10-14T01:07:41.804794+0000","mapping_epoch":13,"log_start":"135'237604","ondisk_log_start":"135'237604","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3282849923,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":248537,"num_read_kb":976877,"num_write":247039,"num_write_kb":988078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"138'279210","reported_seq":559285,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:07:40.858629+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:07:40.858629+0000","last_peered":"2023-10-14T01:07:40.858629+0000","last_clean":"2023-10-14T01:07:40.858629+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:07:40.858629+0000","last_undegraded":"2023-10-14T01:07:40.858629+0000","last_fullsized":"2023-10-14T01:07:40.858629+0000","mapping_epoch":13,"log_start":"136'269207","ondisk_log_start":"136'269207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":686,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 346s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3691368767,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":280725,"num_read_kb":1097610,"num_write":277255,"num_write_kb":1108873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"138'269545","reported_seq":534884,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:07:42.724603+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:07:42.724603+0000","last_peered":"2023-10-14T01:07:42.724603+0000","last_clean":"2023-10-14T01:07:42.724603+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:07:42.724603+0000","last_undegraded":"2023-10-14T01:07:42.724603+0000","last_fullsized":"2023-10-14T01:07:42.724603+0000","mapping_epoch":13,"log_start":"136'259509","ondisk_log_start":"136'259509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":925,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 342s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3464032525,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":266732,"num_read_kb":1030680,"num_write":261370,"num_write_kb":1045266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"138'273445","reported_seq":545498,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:07:42.698744+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:07:42.698744+0000","last_peered":"2023-10-14T01:07:42.698744+0000","last_clean":"2023-10-14T01:07:42.698744+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:07:42.698744+0000","last_undegraded":"2023-10-14T01:07:42.698744+0000","last_fullsized":"2023-10-14T01:07:42.698744+0000","mapping_epoch":13,"log_start":"136'263410","ondisk_log_start":"136'263410","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":870,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 346s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3511820537,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":273699,"num_read_kb":1052380,"num_write":263591,"num_write_kb":1054090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"138'268174","reported_seq":530234,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.844535+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:07:41.844535+0000","last_peered":"2023-10-14T01:07:41.844535+0000","last_clean":"2023-10-14T01:07:41.844535+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:07:41.844535+0000","last_undegraded":"2023-10-14T01:07:41.844535+0000","last_fullsized":"2023-10-14T01:07:41.844535+0000","mapping_epoch":13,"log_start":"136'258107","ondisk_log_start":"136'258107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3449843896,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":263035,"num_read_kb":1026363,"num_write":259586,"num_write_kb":1038231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":344,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:06:54.433013+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:06:54.433013+0000","last_peered":"2023-10-14T01:06:54.433013+0000","last_clean":"2023-10-14T01:06:54.433013+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:06:54.433013+0000","last_undegraded":"2023-10-14T01:06:54.433013+0000","last_fullsized":"2023-10-14T01:06:54.433013+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27771533624,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2129797,"num_read_kb":8272626,"num_write":2089444,"num_write_kb":8356305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17112891392,"data_stored":21424370288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80356,"ondisk_log_size":80356,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739227,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4714436,"kb_used_data":3138492,"kb_used_omap":212,"kb_used_meta":1575723,"kb_avail":89657404,"statfs":{"total":96636764160,"available":91809181696,"internally_reserved":0,"allocated":3213815808,"data_stored":4023012716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1613540596},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:07:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":0.85999999999999999,"15min":0.83099999999999996},"min":{"1min":0.19400000000000001,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":7.492,"5min":7.492,"15min":9.4209999999999994},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.77800000000000002},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":7.415,"5min":7.782,"15min":10.847},"last":2.7050000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:07:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.405,"15min":1.1359999999999999},"min":{"1min":0.246,"5min":0.191,"15min":0.191},"max":{"1min":5.7919999999999998,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":1.121,"15min":1.1439999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":1.3520000000000001,"5min":7.71,"15min":16.736999999999998},"last":0.48599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739215,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10102648,"kb_used_data":8356204,"kb_used_omap":633,"kb_used_meta":1745798,"kb_avail":84269192,"statfs":{"total":96636764160,"available":86291652608,"internally_reserved":0,"allocated":8556752896,"data_stored":10712249862,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":649005,"internal_metadata":1787697363},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,10,3,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:07:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.53000000000000003,"15min":0.80000000000000004},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":2.9300000000000002,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.62,"15min":0.83499999999999996},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":3.1080000000000001,"5min":5.8040000000000003,"15min":12.407},"last":0.53500000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.1140000000000001,"15min":1.1160000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.217},"max":{"1min":2.6190000000000002,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.504},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.3,"15min":1.0780000000000001},"min":{"1min":0.22900000000000001,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":7.9290000000000003,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.56000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739225,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6016184,"kb_used_data":5219184,"kb_used_omap":373,"kb_used_meta":796618,"kb_avail":88355656,"statfs":{"total":96636764160,"available":90476191744,"internally_reserved":0,"allocated":5344444416,"data_stored":6689968968,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":815737558},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,8,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:07:12 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.84099999999999997,"15min":0.995},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":0.60399999999999998,"5min":6.7290000000000001,"15min":12.855},"last":0.246},{"interface":"front","average":{"1min":0.317,"5min":0.52800000000000002,"15min":0.92500000000000004},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.497,"5min":6.944,"15min":16.655000000000001},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:07:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.496,"15min":1.379},"min":{"1min":0.23899999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":10.539999999999999,"5min":10.539999999999999,"15min":10.539999999999999},"last":4.3490000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.68899999999999995,"15min":0.77100000000000002},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.1230000000000002,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5343195136,"data_stored":6688962170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8556646400,"data_stored":10712385848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3213049856,"data_stored":4023022270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:07:44.720 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.720 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.720 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.721 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.721 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.721 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.722 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.722 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:07:44.723 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:07:45.027 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:45.022+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:45.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:45.035 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:45.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:45.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:45.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:45.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:45.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:45.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:45.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:45.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:45.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:45.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:45.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:45.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:45.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:45.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:45.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:45.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:45.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:45.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:45.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:45.334+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:46.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:46.418+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:46.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:46.474+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:46.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:46.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:46.492 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:46.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:46.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:46.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:47.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:47.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:47.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:47.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:47.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:47.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:47.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:47.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:47.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:47.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:47.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:47.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:48.227 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:48.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:07:48.227 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:48.222+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:48.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:48.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:48.261 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:48.254+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:48.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:48.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:48.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:48.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:48.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:48.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:49.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:07:49.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:49.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:49.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:49.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:49.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:49.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:49.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:49.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:49.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:49.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:49.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:49.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:51.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:51.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:51.223 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:51.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:51.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:51.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:51.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:51.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:51.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:51.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:51.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:51.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:51.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:51.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:53.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:53.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:53.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:53.286+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:53.611 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:53.606+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:53.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:53.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:53.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:53.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:53.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:53.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:53.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:53.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:54.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:54.964 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:54.964 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:54.964 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:55.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:55.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:55.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:55.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:55.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:55.090+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:55.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:55.110+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:55.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:55.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:55.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:55.458+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:55.461 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:55.458+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:55.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:55.458+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:55.462 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:55.458+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:56.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:56.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:56.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:56.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:56.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:56.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:56.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:56.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:56.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:56.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:57.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:57.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:57.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:57.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:57.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:57.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:57.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:57.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:57.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:57.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:07:58.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:58.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:07:58.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:58.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:07:58.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:58.010+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:58.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:58.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:07:58.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:58.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:58.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:58.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:58.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:58.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:59.058 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:59.054+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:07:59.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:59.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:07:59.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:59.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:07:59.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:07:59.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:07:59.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:07:59.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:07:59.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:59.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:59.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:59.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:59.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:07:59.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:07:59.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:07:59.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:07:59.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:07:59.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:07:59.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:01.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:01.290+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:01.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:01.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:01.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:01.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:01.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:01.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:01.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:01.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:01.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:01.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:01.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:01.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:01.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:01.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:01.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:02.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:02.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:02.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:02.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:02.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:02.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:02.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.494+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:02.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:08:02.693 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:02.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:02.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:02.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:02.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:02.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:02.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:02.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:03.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:03.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:03.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:03.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:03.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:08:03.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:03.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:03.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:03.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:03.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:04.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:04.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:04.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:04.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:04.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:04.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:04.469 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:04.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:08:04.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:04.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:08:04.504 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:04.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:04.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:04.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:04.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:04.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:04.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:04.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:04.723 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:08:05.462 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:08:05.462 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:08:05.490 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2294,"stamp":"2023-10-14T01:08:04.456682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27806247768,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2138438,"num_read_kb":8306619,"num_write":2097967,"num_write_kb":8392269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80510,"ondisk_log_size":80510,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21203364,"kb_used_data":16781252,"kb_used_omap":1219,"kb_used_meta":4420860,"kb_avail":261912156,"statfs":{"total":289910292480,"available":268198047744,"internally_reserved":0,"allocated":17184002048,"data_stored":21494220474,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1249123,"internal_metadata":4526960797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,5,9,13,19,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17068032,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4215,"num_read_kb":16776,"num_write":4173,"num_write_kb":16692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.895677"},"pg_stats":[{"pgid":"2.7","version":"138'281357","reported_seq":552485,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:08:02.189702+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:08:02.189702+0000","last_peered":"2023-10-14T01:08:02.189702+0000","last_clean":"2023-10-14T01:08:02.189702+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:08:02.189702+0000","last_undegraded":"2023-10-14T01:08:02.189702+0000","last_fullsized":"2023-10-14T01:08:02.189702+0000","mapping_epoch":13,"log_start":"136'271302","ondisk_log_start":"136'271302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3516727734,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":272701,"num_read_kb":1053739,"num_write":264899,"num_write_kb":1059354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"138'281402","reported_seq":553215,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:08:02.219369+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:08:02.219369+0000","last_peered":"2023-10-14T01:08:02.219369+0000","last_clean":"2023-10-14T01:08:02.219369+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:08:02.219369+0000","last_undegraded":"2023-10-14T01:08:02.219369+0000","last_fullsized":"2023-10-14T01:08:02.219369+0000","mapping_epoch":13,"log_start":"136'271309","ondisk_log_start":"136'271309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3509846384,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":274982,"num_read_kb":1052885,"num_write":264954,"num_write_kb":1059443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'254518","reported_seq":506613,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:08:02.218863+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:08:02.218863+0000","last_peered":"2023-10-14T01:08:02.218863+0000","last_clean":"2023-10-14T01:08:02.218863+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:08:02.218863+0000","last_undegraded":"2023-10-14T01:08:02.218863+0000","last_fullsized":"2023-10-14T01:08:02.218863+0000","mapping_epoch":13,"log_start":"136'244505","ondisk_log_start":"136'244505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3357925778,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":252533,"num_read_kb":994616,"num_write":253900,"num_write_kb":1015570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"138'248604","reported_seq":497663,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:08:02.206131+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:08:02.206131+0000","last_peered":"2023-10-14T01:08:02.206131+0000","last_clean":"2023-10-14T01:08:02.206131+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:08:02.206131+0000","last_undegraded":"2023-10-14T01:08:02.206131+0000","last_fullsized":"2023-10-14T01:08:02.206131+0000","mapping_epoch":13,"log_start":"136'238504","ondisk_log_start":"136'238504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3286761603,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":249533,"num_read_kb":980833,"num_write":247999,"num_write_kb":991918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"138'280411","reported_seq":561650,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:01.005743+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:08:01.005743+0000","last_peered":"2023-10-14T01:08:01.005743+0000","last_clean":"2023-10-14T01:08:01.005743+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:08:01.005743+0000","last_undegraded":"2023-10-14T01:08:01.005743+0000","last_fullsized":"2023-10-14T01:08:01.005743+0000","mapping_epoch":13,"log_start":"136'270407","ondisk_log_start":"136'270407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":708,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 366s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3696214335,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":281881,"num_read_kb":1102214,"num_write":278440,"num_write_kb":1113613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"138'270661","reported_seq":537106,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:03.037570+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:08:03.037570+0000","last_peered":"2023-10-14T01:08:03.037570+0000","last_clean":"2023-10-14T01:08:03.037570+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:08:03.037570+0000","last_undegraded":"2023-10-14T01:08:03.037570+0000","last_fullsized":"2023-10-14T01:08:03.037570+0000","mapping_epoch":13,"log_start":"136'260609","ondisk_log_start":"136'260609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":985,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 362s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3468222733,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":267819,"num_read_kb":1035004,"num_write":262398,"num_write_kb":1049378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"138'274560","reported_seq":547725,"reported_epoch":138,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:02.941698+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:08:02.941698+0000","last_peered":"2023-10-14T01:08:02.941698+0000","last_clean":"2023-10-14T01:08:02.941698+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:08:02.941698+0000","last_undegraded":"2023-10-14T01:08:02.941698+0000","last_fullsized":"2023-10-14T01:08:02.941698+0000","mapping_epoch":13,"log_start":"136'264511","ondisk_log_start":"136'264511","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":922,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 366s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3515977977,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":274779,"num_read_kb":1056680,"num_write":264606,"num_write_kb":1058150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"138'269224","reported_seq":532345,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:08:02.195803+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:08:02.195803+0000","last_peered":"2023-10-14T01:08:02.195803+0000","last_clean":"2023-10-14T01:08:02.195803+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:08:02.195803+0000","last_undegraded":"2023-10-14T01:08:02.195803+0000","last_fullsized":"2023-10-14T01:08:02.195803+0000","mapping_epoch":13,"log_start":"136'259207","ondisk_log_start":"136'259207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3453980856,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":264084,"num_read_kb":1030539,"num_write":260602,"num_write_kb":1042295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":344,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:06:54.433013+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:06:54.433013+0000","last_peered":"2023-10-14T01:06:54.433013+0000","last_clean":"2023-10-14T01:06:54.433013+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:06:54.433013+0000","last_undegraded":"2023-10-14T01:06:54.433013+0000","last_fullsized":"2023-10-14T01:06:54.433013+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27805657400,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2138312,"num_read_kb":8306510,"num_write":2097798,"num_write_kb":8389721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17181216768,"data_stored":21492695664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34433,"internal_metadata":0},"log_size":80383,"ondisk_log_size":80383,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739231,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4758596,"kb_used_data":3150716,"kb_used_omap":212,"kb_used_meta":1607659,"kb_avail":89613244,"statfs":{"total":96636764160,"available":91763961856,"internally_reserved":0,"allocated":3226333184,"data_stored":4035530092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1646243060},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:07:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":0.85999999999999999,"15min":0.83099999999999996},"min":{"1min":0.19400000000000001,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":7.492,"5min":7.492,"15min":9.4209999999999994},"last":0.377},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.78200000000000003,"15min":0.77800000000000002},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":7.415,"5min":7.782,"15min":10.847},"last":0.42199999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:07:06 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.405,"15min":1.1359999999999999},"min":{"1min":0.246,"5min":0.191,"15min":0.191},"max":{"1min":5.7919999999999998,"5min":14.103999999999999,"15min":14.103999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":1.121,"15min":1.1439999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.19},"max":{"1min":1.3520000000000001,"5min":7.71,"15min":16.736999999999998},"last":0.499}]}]},{"osd":1,"up_from":8,"seq":34359739219,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10345416,"kb_used_data":8390652,"kb_used_omap":633,"kb_used_meta":1954118,"kb_avail":84026424,"statfs":{"total":96636764160,"available":86043058176,"internally_reserved":0,"allocated":8592027648,"data_stored":10747524614,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":649005,"internal_metadata":2001017043},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,3,6,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:07:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.53000000000000003,"15min":0.80000000000000004},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":2.9300000000000002,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.62,"15min":0.83499999999999996},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":3.1080000000000001,"5min":5.8040000000000003,"15min":12.407},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.1140000000000001,"15min":1.1160000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.217},"max":{"1min":2.6190000000000002,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.435},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.3,"15min":1.0780000000000001},"min":{"1min":0.22900000000000001,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":7.9290000000000003,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.35299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6099352,"kb_used_data":5239884,"kb_used_omap":373,"kb_used_meta":859082,"kb_avail":88272488,"statfs":{"total":96636764160,"available":90391027712,"internally_reserved":0,"allocated":5365641216,"data_stored":6711165768,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":879700694},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:07:12 2023","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.84099999999999997,"15min":0.995},"min":{"1min":0.214,"5min":0.17699999999999999,"15min":0.16800000000000001},"max":{"1min":0.60399999999999998,"5min":6.7290000000000001,"15min":12.855},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.317,"5min":0.52800000000000002,"15min":0.92500000000000004},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.497,"5min":6.944,"15min":16.655000000000001},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:07:12 2023","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.496,"15min":1.379},"min":{"1min":0.23899999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":10.539999999999999,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.68899999999999995,"15min":0.77100000000000002},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.1230000000000002,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.33200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5364649984,"data_stored":6710417018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8590987264,"data_stored":10746726712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15537,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3225579520,"data_stored":4035551934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:08:05.494 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.494 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.495 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.495 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.495 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.496 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.496 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.497 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:05.497 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:08:07.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:07.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:08:07.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:07.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:07.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:07.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:07.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:07.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:07.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:07.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:07.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:07.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:07.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:07.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:09.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:09.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:09.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:09.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:09.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:09.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:09.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:09.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:09.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:09.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:09.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:09.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:09.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:09.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:10.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:10.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:10.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:10.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:10.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.122+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:10.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:10.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:10.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:10.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:10.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:10.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:10.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:10.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:10.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:11.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:11.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:11.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:11.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:11.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:11.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:11.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:11.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:11.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:11.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:11.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:11.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:11.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:11.986+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:12.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:12.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:12.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:12.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.618+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:12.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:12.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:12.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:12.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:12.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:12.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:12.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:08:12.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:12.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:12.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:12.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:12.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:12.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:12.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:13.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:13.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:08:13.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:13.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:08:13.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:13.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:08:13.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:13.062+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:13.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:13.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:13.085 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:13.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:13.085 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:13.082+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:14.157 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:14.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:14.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:14.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:14.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:14.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:14.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:14.178+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:14.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:14.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:14.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:14.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:15.183 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:15.177+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:08:15.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:15.197+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:16.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:16.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:16.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:16.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:16.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:16.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:16.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:16.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:16.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:17.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:17.733+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:18.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:18.873+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:18.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:18.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:18.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:18.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:19.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:18.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:19.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:19.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:19.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:19.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:19.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:19.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:21.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:21.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:21.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:21.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:21.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:21.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:21.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:21.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:21.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:21.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:21.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:25.498 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:08:25.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:25.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:25.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:25.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:26.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:25.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:26.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:26.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:26.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:08:26.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:08:26.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:26.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:26.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:26.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:26.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:26.341 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:08:26.357 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:08:26.410 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2305,"stamp":"2023-10-14T01:08:25.566289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27840473944,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2146952,"num_read_kb":8340495,"num_write":2106327,"num_write_kb":8425709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80527,"ondisk_log_size":80527,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21189748,"kb_used_data":16847320,"kb_used_omap":1209,"kb_used_meta":4341190,"kb_avail":261925772,"statfs":{"total":289910292480,"available":268211990528,"internally_reserved":0,"allocated":17251655680,"data_stored":21561851153,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1238605,"internal_metadata":4445378995},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,2,10,14,17,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17358848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4412,"num_read_kb":17552,"num_write":4248,"num_write_kb":16992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001942"},"pg_stats":[{"pgid":"2.7","version":"139'282497","reported_seq":554749,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:22.483940+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:08:22.483940+0000","last_peered":"2023-10-14T01:08:22.483940+0000","last_clean":"2023-10-14T01:08:22.483940+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:08:22.483940+0000","last_undegraded":"2023-10-14T01:08:22.483940+0000","last_fullsized":"2023-10-14T01:08:22.483940+0000","mapping_epoch":13,"log_start":"136'272402","ondisk_log_start":"136'272402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3521089974,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":273805,"num_read_kb":1058123,"num_write":265959,"num_write_kb":1063594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'282516","reported_seq":555431,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:22.470094+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:08:22.470094+0000","last_peered":"2023-10-14T01:08:22.470094+0000","last_clean":"2023-10-14T01:08:22.470094+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:08:22.470094+0000","last_undegraded":"2023-10-14T01:08:22.470094+0000","last_fullsized":"2023-10-14T01:08:22.470094+0000","mapping_epoch":13,"log_start":"136'272509","ondisk_log_start":"136'272509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3514102128,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":276067,"num_read_kb":1057197,"num_write":265989,"num_write_kb":1063583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"139'255538","reported_seq":508668,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:22.467296+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:08:22.467296+0000","last_peered":"2023-10-14T01:08:22.467296+0000","last_clean":"2023-10-14T01:08:22.467296+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:08:22.467296+0000","last_undegraded":"2023-10-14T01:08:22.467296+0000","last_fullsized":"2023-10-14T01:08:22.467296+0000","mapping_epoch":13,"log_start":"136'245505","ondisk_log_start":"136'245505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3362111890,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":253564,"num_read_kb":998708,"num_write":254920,"num_write_kb":1019650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'249591","reported_seq":499670,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:22.442863+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:08:22.442863+0000","last_peered":"2023-10-14T01:08:22.442863+0000","last_clean":"2023-10-14T01:08:22.442863+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:08:22.442863+0000","last_undegraded":"2023-10-14T01:08:22.442863+0000","last_fullsized":"2023-10-14T01:08:22.442863+0000","mapping_epoch":13,"log_start":"136'239504","ondisk_log_start":"136'239504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3290796163,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":250553,"num_read_kb":984885,"num_write":248986,"num_write_kb":995866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"139'281509","reported_seq":563873,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:21.137716+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:08:21.137716+0000","last_peered":"2023-10-14T01:08:21.137716+0000","last_clean":"2023-10-14T01:08:21.137716+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:08:21.137716+0000","last_undegraded":"2023-10-14T01:08:21.137716+0000","last_fullsized":"2023-10-14T01:08:21.137716+0000","mapping_epoch":13,"log_start":"136'271507","ondisk_log_start":"136'271507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":769,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 386s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3700551999,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":282989,"num_read_kb":1106622,"num_write":279503,"num_write_kb":1117865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"139'271789","reported_seq":539322,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:23.246732+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:08:23.246732+0000","last_peered":"2023-10-14T01:08:23.246732+0000","last_clean":"2023-10-14T01:08:23.246732+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:08:23.246732+0000","last_undegraded":"2023-10-14T01:08:23.246732+0000","last_fullsized":"2023-10-14T01:08:23.246732+0000","mapping_epoch":13,"log_start":"136'261709","ondisk_log_start":"136'261709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1042,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 383s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3472580877,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":268872,"num_read_kb":1039216,"num_write":263467,"num_write_kb":1053654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"139'275669","reported_seq":549942,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:23.246588+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:08:23.246588+0000","last_peered":"2023-10-14T01:08:23.246588+0000","last_clean":"2023-10-14T01:08:23.246588+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:08:23.246588+0000","last_undegraded":"2023-10-14T01:08:23.246588+0000","last_fullsized":"2023-10-14T01:08:23.246588+0000","mapping_epoch":13,"log_start":"136'265611","ondisk_log_start":"136'265611","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":974,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 387s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3520254201,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":275856,"num_read_kb":1060968,"num_write":265651,"num_write_kb":1062330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"139'270345","reported_seq":534511,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:22.462144+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:08:22.462144+0000","last_peered":"2023-10-14T01:08:22.462144+0000","last_clean":"2023-10-14T01:08:22.462144+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:08:22.462144+0000","last_undegraded":"2023-10-14T01:08:22.462144+0000","last_fullsized":"2023-10-14T01:08:22.462144+0000","mapping_epoch":13,"log_start":"136'260307","ondisk_log_start":"136'260307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3458396344,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":265120,"num_read_kb":1034667,"num_write":261683,"num_write_kb":1046619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":345,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.832064+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:07:41.832064+0000","last_peered":"2023-10-14T01:07:41.832064+0000","last_clean":"2023-10-14T01:07:41.832064+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:07:41.832064+0000","last_undegraded":"2023-10-14T01:07:41.832064+0000","last_fullsized":"2023-10-14T01:07:41.832064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27839883576,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2146826,"num_read_kb":8340386,"num_write":2106158,"num_write_kb":8423161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17249509376,"data_stored":21560988272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80400,"ondisk_log_size":80400,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739235,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4803204,"kb_used_data":3163388,"kb_used_omap":212,"kb_used_meta":1639595,"kb_avail":89568636,"statfs":{"total":96636764160,"available":91718283264,"internally_reserved":0,"allocated":3239309312,"data_stored":4048498569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1678945524},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.95499999999999996,"15min":0.876},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":5.5860000000000003,"5min":7.492,"15min":9.4209999999999994},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.2010000000000001,"5min":0.81799999999999995,"15min":0.83099999999999996},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":6.6539999999999999,"5min":7.782,"15min":10.847},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":1.2310000000000001,"15min":1.1339999999999999},"min":{"1min":0.23000000000000001,"5min":0.191,"15min":0.191},"max":{"1min":2.5590000000000002,"5min":9.657,"15min":14.103999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.129,"15min":1.161},"min":{"1min":0.23699999999999999,"5min":0.21099999999999999,"15min":0.19},"max":{"1min":5.335,"5min":7.71,"15min":16.736999999999998},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739223,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10190336,"kb_used_data":8423544,"kb_used_omap":623,"kb_used_meta":1766160,"kb_avail":84181504,"statfs":{"total":96636764160,"available":86201860096,"internally_reserved":0,"allocated":8625709056,"data_stored":10781198371,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1808548329},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,7,7,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:07:21 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.53000000000000003,"15min":0.80000000000000004},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":2.9300000000000002,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.62,"15min":0.83499999999999996},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":3.1080000000000001,"5min":5.8040000000000003,"15min":12.407},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:07:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.1140000000000001,"15min":1.1160000000000001},"min":{"1min":0.23699999999999999,"5min":0.222,"15min":0.217},"max":{"1min":2.6190000000000002,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.3,"15min":1.0780000000000001},"min":{"1min":0.22900000000000001,"5min":0.20499999999999999,"15min":0.20200000000000001},"max":{"1min":7.9290000000000003,"5min":39.412999999999997,"15min":39.412999999999997},"last":1.3700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6196208,"kb_used_data":5260388,"kb_used_omap":373,"kb_used_meta":935434,"kb_avail":88175632,"statfs":{"total":96636764160,"available":90291847168,"internally_reserved":0,"allocated":5386637312,"data_stored":6732154213,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":957885142},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,6,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:08:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69499999999999995,"15min":0.98499999999999999},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":3.6859999999999999,"5min":6.7290000000000001,"15min":12.855},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.131,"5min":0.55900000000000005,"15min":0.92400000000000004},"min":{"1min":0.22600000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":9.2439999999999998,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:08:15 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.5089999999999999,"15min":1.385},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":9.3460000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.63900000000000001,"15min":0.76700000000000002},"min":{"1min":0.23499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.3500000000000001,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5385646080,"data_stored":6731413114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8625172480,"data_stored":10780911928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3238690816,"data_stored":4048663230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:08:26.411 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.412 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.412 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.413 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.413 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.414 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.414 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.414 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:26.415 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:08:28.504 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:28.497+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:28.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:28.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:28.557 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:28.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:28.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:28.553+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:31.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:31.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:31.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:31.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:31.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:31.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:31.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:31.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:31.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:31.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:31.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:31.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:31.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:31.973+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:08:31.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:31.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:31.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:31.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:31.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:31.985+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:32.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:32.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:32.084 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:32.077+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:32.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:32.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:32.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:32.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:32.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:32.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:32.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:32.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:32.813+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:08:32.817 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:32.813+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:08:32.874 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:32.869+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:32.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:32.901+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:32.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:32.901+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:32.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:32.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:32.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:32.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:32.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:32.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:33.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:33.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:33.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:33.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:33.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:33.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:33.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:33.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:33.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:33.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:35.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:08:35.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:08:35.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:08:35.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:35.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:35.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:35.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:35.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:35.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:35.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:35.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:37.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:37.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:37.050 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:37.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:37.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:37.049+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:37.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:37.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:37.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:37.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:37.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:37.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:37.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:37.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:37.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:37.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:40.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:40.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:40.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:40.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:40.993 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:40.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:08:41.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.001+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:08:41.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:41.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:41.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:41.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:41.377 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.377+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:08:41.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:41.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:41.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.441+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:41.454 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:41.453+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:43.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:43.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:43.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:43.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:43.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:43.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:43.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:43.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:43.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:43.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:44.504 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:44.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:08:44.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:44.521+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:45.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:45.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:45.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:45.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:45.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:45.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:45.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:45.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:45.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:45.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:45.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:45.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:45.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:45.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:08:45.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:45.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:45.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:45.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:45.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:45.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:46.416 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:08:46.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:46.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:08:46.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:46.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:46.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:46.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:46.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:46.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:46.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:46.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:46.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:46.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:08:47.268 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:08:47.269 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:08:47.351 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2315,"stamp":"2023-10-14T01:08:45.574674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27874691928,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2155713,"num_read_kb":8375331,"num_write":2114706,"num_write_kb":8459225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80474,"ondisk_log_size":80474,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21453344,"kb_used_data":16913668,"kb_used_omap":1209,"kb_used_meta":4538438,"kb_avail":261662176,"statfs":{"total":289910292480,"available":267942068224,"internally_reserved":0,"allocated":17319596032,"data_stored":21629791505,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1238605,"internal_metadata":4647360947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,1,6,8,10,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17158144,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4450,"num_read_kb":17672,"num_write":4202,"num_write_kb":16808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001971"},"pg_stats":[{"pgid":"2.7","version":"139'283649","reported_seq":557026,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:42.907121+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:08:42.907121+0000","last_peered":"2023-10-14T01:08:42.907121+0000","last_clean":"2023-10-14T01:08:42.907121+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:08:42.907121+0000","last_undegraded":"2023-10-14T01:08:42.907121+0000","last_fullsized":"2023-10-14T01:08:42.907121+0000","mapping_epoch":13,"log_start":"137'273602","ondisk_log_start":"137'273602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3525603766,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":274913,"num_read_kb":1062515,"num_write":267067,"num_write_kb":1068026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'283614","reported_seq":557625,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:42.894785+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:08:42.894785+0000","last_peered":"2023-10-14T01:08:42.894785+0000","last_clean":"2023-10-14T01:08:42.894785+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:08:42.894785+0000","last_undegraded":"2023-10-14T01:08:42.894785+0000","last_fullsized":"2023-10-14T01:08:42.894785+0000","mapping_epoch":13,"log_start":"137'273609","ondisk_log_start":"137'273609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3518382448,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":277146,"num_read_kb":1061473,"num_write":267047,"num_write_kb":1067815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"139'256584","reported_seq":510793,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:42.893696+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:08:42.893696+0000","last_peered":"2023-10-14T01:08:42.893696+0000","last_clean":"2023-10-14T01:08:42.893696+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:08:42.893696+0000","last_undegraded":"2023-10-14T01:08:42.893696+0000","last_fullsized":"2023-10-14T01:08:42.893696+0000","mapping_epoch":13,"log_start":"136'246505","ondisk_log_start":"136'246505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3366404498,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":254644,"num_read_kb":1002996,"num_write":255966,"num_write_kb":1023834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'250576","reported_seq":501723,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:42.851689+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:08:42.851689+0000","last_peered":"2023-10-14T01:08:42.851689+0000","last_clean":"2023-10-14T01:08:42.851689+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:08:42.851689+0000","last_undegraded":"2023-10-14T01:08:42.851689+0000","last_fullsized":"2023-10-14T01:08:42.851689+0000","mapping_epoch":13,"log_start":"136'240504","ondisk_log_start":"136'240504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3294834819,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":251620,"num_read_kb":989133,"num_write":249971,"num_write_kb":999806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"139'282619","reported_seq":566124,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:41.338144+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:08:41.338144+0000","last_peered":"2023-10-14T01:08:41.338144+0000","last_clean":"2023-10-14T01:08:41.338144+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:08:41.338144+0000","last_undegraded":"2023-10-14T01:08:41.338144+0000","last_fullsized":"2023-10-14T01:08:41.338144+0000","mapping_epoch":13,"log_start":"137'272607","ondisk_log_start":"137'272607","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":829,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 406s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3704938815,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":284115,"num_read_kb":1111102,"num_write":280577,"num_write_kb":1122161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"139'272861","reported_seq":541467,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:43.718667+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:08:43.718667+0000","last_peered":"2023-10-14T01:08:43.718667+0000","last_clean":"2023-10-14T01:08:43.718667+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:08:43.718667+0000","last_undegraded":"2023-10-14T01:08:43.718667+0000","last_fullsized":"2023-10-14T01:08:43.718667+0000","mapping_epoch":13,"log_start":"136'262809","ondisk_log_start":"136'262809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1092,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 403s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3476721933,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":269917,"num_read_kb":1043380,"num_write":264480,"num_write_kb":1057706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"139'276820","reported_seq":552253,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:08:43.703809+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:08:43.703809+0000","last_peered":"2023-10-14T01:08:43.703809+0000","last_clean":"2023-10-14T01:08:43.703809+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:08:43.703809+0000","last_undegraded":"2023-10-14T01:08:43.703809+0000","last_fullsized":"2023-10-14T01:08:43.703809+0000","mapping_epoch":13,"log_start":"137'266811","ondisk_log_start":"137'266811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1022,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 407s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3524686073,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":276986,"num_read_kb":1065468,"num_write":266735,"num_write_kb":1066666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"139'271378","reported_seq":536679,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:42.854093+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:08:42.854093+0000","last_peered":"2023-10-14T01:08:42.854093+0000","last_clean":"2023-10-14T01:08:42.854093+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:08:42.854093+0000","last_undegraded":"2023-10-14T01:08:42.854093+0000","last_fullsized":"2023-10-14T01:08:42.854093+0000","mapping_epoch":13,"log_start":"136'261307","ondisk_log_start":"136'261307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3462529208,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":266246,"num_read_kb":1039155,"num_write":262694,"num_write_kb":1050663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":345,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.832064+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:07:41.832064+0000","last_peered":"2023-10-14T01:07:41.832064+0000","last_clean":"2023-10-14T01:07:41.832064+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:07:41.832064+0000","last_undegraded":"2023-10-14T01:07:41.832064+0000","last_fullsized":"2023-10-14T01:07:41.832064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27874101560,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2155587,"num_read_kb":8375222,"num_write":2114537,"num_write_kb":8456677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17318223872,"data_stored":21629702768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80347,"ondisk_log_size":80347,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739239,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4847584,"kb_used_data":3175832,"kb_used_omap":212,"kb_used_meta":1671531,"kb_avail":89524256,"statfs":{"total":96636764160,"available":91672838144,"internally_reserved":0,"allocated":3252051968,"data_stored":4061241225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1711647988},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:08:09 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":0.95499999999999996,"15min":0.876},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.14699999999999999},"max":{"1min":5.5860000000000003,"5min":7.492,"15min":9.4209999999999994},"last":0.22},{"interface":"front","average":{"1min":1.2010000000000001,"5min":0.81799999999999995,"15min":0.83099999999999996},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":6.6539999999999999,"5min":7.782,"15min":10.847},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:08:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":1.2310000000000001,"15min":1.1339999999999999},"min":{"1min":0.23000000000000001,"5min":0.191,"15min":0.191},"max":{"1min":2.5590000000000002,"5min":9.657,"15min":14.103999999999999},"last":7.9589999999999996},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.129,"15min":1.161},"min":{"1min":0.23699999999999999,"5min":0.21099999999999999,"15min":0.19},"max":{"1min":5.335,"5min":7.71,"15min":16.736999999999998},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359739227,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10330444,"kb_used_data":8456640,"kb_used_omap":623,"kb_used_meta":1873168,"kb_avail":84041396,"statfs":{"total":96636764160,"available":86058389504,"internally_reserved":0,"allocated":8659599360,"data_stored":10815088675,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1918124521},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:08:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.51500000000000001,"15min":0.77900000000000003},"min":{"1min":0.20200000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.55600000000000005,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.61899999999999999,"15min":0.82199999999999995},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":1.3939999999999999,"5min":5.8040000000000003,"15min":12.407},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.083,"15min":1.1020000000000001},"min":{"1min":0.246,"5min":0.222,"15min":0.22},"max":{"1min":3.5739999999999998,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.2769999999999999,"15min":1.0880000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":3.2469999999999999,"5min":39.412999999999997,"15min":39.412999999999997},"last":3.1549999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739237,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6275316,"kb_used_data":5281196,"kb_used_omap":373,"kb_used_meta":993738,"kb_avail":88096524,"statfs":{"total":96636764160,"available":90210840576,"internally_reserved":0,"allocated":5407944704,"data_stored":6753461605,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1017588438},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:08:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69499999999999995,"15min":0.98499999999999999},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":3.6859999999999999,"5min":6.7290000000000001,"15min":12.855},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.131,"5min":0.55900000000000005,"15min":0.92400000000000004},"min":{"1min":0.22600000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":9.2439999999999998,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.312}]},{"osd":2,"last update":"Sat Oct 14 01:08:15 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.5089999999999999,"15min":1.385},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":9.3460000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.63900000000000001,"15min":0.76700000000000002},"min":{"1min":0.23499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.3500000000000001,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.32500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5407117312,"data_stored":6752884346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8659689472,"data_stored":10815428920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3251417088,"data_stored":4061389502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:08:47.352 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.352 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.353 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.353 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.353 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.354 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.354 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.355 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:08:47.355 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:08:48.256 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:08:48.307 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-14T01:08:48.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:48.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:49.121 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-14T01:08:49.154 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:08:49.995 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:08:50.050 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-14T01:08:50.762 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-14T01:08:50.812 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-10-14T01:08:51.616 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-10-14T01:08:51.665 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-10-14T01:08:52.398 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-10-14T01:08:52.449 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-14T01:08:52.607 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:52.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:08:52.607 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:52.605+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:08:52.613 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:52.613+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:08:52.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:52.617+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:52.660 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:52.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:08:52.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:52.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:08:52.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:52.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:08:53.375 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-14T01:08:53.435 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-14T01:08:54.485 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-14T01:08:54.522 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:08:54.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:08:54.541+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:08:55.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:55.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:08:55.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:55.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:08:55.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:55.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:08:55.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:55.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:08:55.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:08:55.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:00.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:00.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:00.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:00.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:00.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:00.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:00.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:00.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:01.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:01.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:01.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:01.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:01.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:01.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:01.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:01.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:01.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:02.661 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:02.657+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:03.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:03.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:03.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:03.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:03.725 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:03.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:03.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:03.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:03.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:03.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:03.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:03.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:06.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:06.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:06.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:06.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:06.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:06.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:06.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:06.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:06.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:06.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:06.734 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:06.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:06.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:07.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:07.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:09:07.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:07.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:07.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:07.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:07.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:07.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:07.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:07.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:07.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:07.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:07.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:07.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:08.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:08.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:08.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:08.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:08.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:08.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:08.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:08.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:08.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:08.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:08.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:08.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:08.928 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:08.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:08.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:08.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:08.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:08.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:11.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:11.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:11.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:11.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:11.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:11.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:11.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:11.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:11.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:11.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:11.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:11.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:12.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:12.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:12.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:12.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:12.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:12.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:12.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:12.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:12.082 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:12.388 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:12.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:12.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:12.452+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:12.784 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:12.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:12.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:12.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:12.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:12.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:12.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:12.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:12.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:09:12.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:12.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:12.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:12.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:12.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:12.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:12.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:12.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:12.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:12.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:13.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:09:13.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:13.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:13.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:13.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:13.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:13.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:13.554 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:13.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:13.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:13.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:14.524 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:09:15.429 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:09:15.429 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:09:15.481 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2330,"stamp":"2023-10-14T01:09:14.483008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27924765528,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2168499,"num_read_kb":8426127,"num_write":2126943,"num_write_kb":8508173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80467,"ondisk_log_size":80467,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20892912,"kb_used_data":17012692,"kb_used_omap":1209,"kb_used_meta":3878982,"kb_avail":262222608,"statfs":{"total":289910292480,"available":268515950592,"internally_reserved":0,"allocated":17420996608,"data_stored":21731192081,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1238605,"internal_metadata":3972078003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,3,11,14,17,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19271680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4743,"num_read_kb":18840,"num_write":4705,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.905550"},"pg_stats":[{"pgid":"2.7","version":"139'285208","reported_seq":560197,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:09:13.416890+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:09:13.416890+0000","last_peered":"2023-10-14T01:09:13.416890+0000","last_clean":"2023-10-14T01:09:13.416890+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:09:13.416890+0000","last_undegraded":"2023-10-14T01:09:13.416890+0000","last_fullsized":"2023-10-14T01:09:13.416890+0000","mapping_epoch":13,"log_start":"137'275202","ondisk_log_start":"137'275202","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3531555254,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":276504,"num_read_kb":1068839,"num_write":268532,"num_write_kb":1073886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"139'285251","reported_seq":560843,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:09:13.497760+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:09:13.497760+0000","last_peered":"2023-10-14T01:09:13.497760+0000","last_clean":"2023-10-14T01:09:13.497760+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:09:13.497760+0000","last_undegraded":"2023-10-14T01:09:13.497760+0000","last_fullsized":"2023-10-14T01:09:13.497760+0000","mapping_epoch":13,"log_start":"137'275209","ondisk_log_start":"137'275209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3524718960,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":278705,"num_read_kb":1067665,"num_write":268594,"num_write_kb":1074003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"139'258032","reported_seq":513818,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:09:13.480021+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:09:13.480021+0000","last_peered":"2023-10-14T01:09:13.480021+0000","last_clean":"2023-10-14T01:09:13.480021+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:09:13.480021+0000","last_undegraded":"2023-10-14T01:09:13.480021+0000","last_fullsized":"2023-10-14T01:09:13.480021+0000","mapping_epoch":13,"log_start":"137'248005","ondisk_log_start":"137'248005","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3372310930,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":256218,"num_read_kb":1009240,"num_write":257414,"num_write_kb":1029626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"139'252000","reported_seq":504684,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:09:13.459818+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:09:13.459818+0000","last_peered":"2023-10-14T01:09:13.459818+0000","last_clean":"2023-10-14T01:09:13.459818+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:09:13.459818+0000","last_undegraded":"2023-10-14T01:09:13.459818+0000","last_fullsized":"2023-10-14T01:09:13.459818+0000","mapping_epoch":13,"log_start":"137'241904","ondisk_log_start":"137'241904","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3300696195,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":253155,"num_read_kb":995233,"num_write":251395,"num_write_kb":1005502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"139'284368","reported_seq":569617,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:11.654401+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:09:11.654401+0000","last_peered":"2023-10-14T01:09:11.654401+0000","last_clean":"2023-10-14T01:09:11.654401+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:09:11.654401+0000","last_undegraded":"2023-10-14T01:09:11.654401+0000","last_fullsized":"2023-10-14T01:09:11.654401+0000","mapping_epoch":13,"log_start":"137'274307","ondisk_log_start":"137'274307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":886,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 437s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3711951167,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":285835,"num_read_kb":1117922,"num_write":282288,"num_write_kb":1129005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"139'274434","reported_seq":544647,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:14.137656+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:09:14.137656+0000","last_peered":"2023-10-14T01:09:14.137656+0000","last_clean":"2023-10-14T01:09:14.137656+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:09:14.137656+0000","last_undegraded":"2023-10-14T01:09:14.137656+0000","last_fullsized":"2023-10-14T01:09:14.137656+0000","mapping_epoch":13,"log_start":"137'264409","ondisk_log_start":"137'264409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1165,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 434s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3482796301,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":271489,"num_read_kb":1049632,"num_write":265966,"num_write_kb":1063650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"139'278480","reported_seq":555546,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:14.156327+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:09:14.156327+0000","last_peered":"2023-10-14T01:09:14.156327+0000","last_clean":"2023-10-14T01:09:14.156327+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:09:14.156327+0000","last_undegraded":"2023-10-14T01:09:14.156327+0000","last_fullsized":"2023-10-14T01:09:14.156327+0000","mapping_epoch":13,"log_start":"137'268411","ondisk_log_start":"137'268411","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1098,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 437s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3531051257,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":278582,"num_read_kb":1071824,"num_write":268293,"num_write_kb":1072898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"139'273021","reported_seq":539973,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:09:13.453660+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:09:13.453660+0000","last_peered":"2023-10-14T01:09:13.453660+0000","last_clean":"2023-10-14T01:09:13.453660+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:09:13.453660+0000","last_undegraded":"2023-10-14T01:09:13.453660+0000","last_fullsized":"2023-10-14T01:09:13.453660+0000","mapping_epoch":13,"log_start":"137'263007","ondisk_log_start":"137'263007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3469095096,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":267885,"num_read_kb":1045663,"num_write":264292,"num_write_kb":1057055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":345,"reported_epoch":138,"state":"active+clean","last_fresh":"2023-10-14T01:07:41.832064+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:07:41.832064+0000","last_peered":"2023-10-14T01:07:41.832064+0000","last_clean":"2023-10-14T01:07:41.832064+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:07:41.832064+0000","last_undegraded":"2023-10-14T01:07:41.832064+0000","last_fullsized":"2023-10-14T01:07:41.832064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27924175160,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2168373,"num_read_kb":8426018,"num_write":2126774,"num_write_kb":8505625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17418489856,"data_stored":21729968752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80340,"ondisk_log_size":80340,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739245,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4924588,"kb_used_data":3194528,"kb_used_omap":212,"kb_used_meta":1729835,"kb_avail":89447252,"statfs":{"total":96636764160,"available":91593986048,"internally_reserved":0,"allocated":3271196672,"data_stored":4080385929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1771351284},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.94899999999999995,"15min":0.84599999999999997},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.14699999999999999},"max":{"1min":2.673,"5min":7.492,"15min":9.4209999999999994},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82599999999999996,"15min":0.81499999999999995},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":4.3579999999999997,"5min":7.782,"15min":10.847},"last":0.5}]},{"osd":1,"last update":"Sat Oct 14 01:09:09 2023","interfaces":[{"interface":"back","average":{"1min":2.109,"5min":1.532,"15min":1.23},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":7.9589999999999996,"5min":9.657,"15min":14.103999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.198,"15min":1.208},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.19},"max":{"1min":8.9420000000000002,"5min":8.9420000000000002,"15min":16.736999999999998},"last":0.46600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739233,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9567928,"kb_used_data":8506480,"kb_used_omap":623,"kb_used_meta":1060816,"kb_avail":84803912,"statfs":{"total":96636764160,"available":86839205888,"internally_reserved":0,"allocated":8710635520,"data_stored":10866124835,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1086276073},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,6,9,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:08:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.51500000000000001,"15min":0.77900000000000003},"min":{"1min":0.20200000000000001,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":0.55600000000000005,"5min":4.6180000000000003,"15min":12.369999999999999},"last":0.307},{"interface":"front","average":{"1min":0.443,"5min":0.61899999999999999,"15min":0.82199999999999995},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":1.3939999999999999,"5min":5.8040000000000003,"15min":12.407},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.083,"15min":1.1020000000000001},"min":{"1min":0.246,"5min":0.222,"15min":0.22},"max":{"1min":3.5739999999999998,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.371},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.2769999999999999,"15min":1.0880000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":3.2469999999999999,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.34499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739243,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6400396,"kb_used_data":5311684,"kb_used_omap":373,"kb_used_meta":1088330,"kb_avail":87971444,"statfs":{"total":96636764160,"available":90082758656,"internally_reserved":0,"allocated":5439164416,"data_stored":6784681317,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1114450646},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:08:15 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.69499999999999995,"15min":0.98499999999999999},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":3.6859999999999999,"5min":6.7290000000000001,"15min":12.855},"last":0.505},{"interface":"front","average":{"1min":1.131,"5min":0.55900000000000005,"15min":0.92400000000000004},"min":{"1min":0.22600000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":9.2439999999999998,"5min":9.2439999999999998,"15min":16.655000000000001},"last":2.4950000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:08:15 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.5089999999999999,"15min":1.385},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.17599999999999999},"max":{"1min":9.3460000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.63900000000000001,"15min":0.76700000000000002},"min":{"1min":0.23499999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.3500000000000001,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5438152704,"data_stored":6783919738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8709918720,"data_stored":10865658168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3270418432,"data_stored":4080390846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:09:15.483 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.483 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.484 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.484 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.484 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.485 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.485 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.485 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:15.486 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:09:17.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:17.588 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:17.588+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:17.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:17.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:17.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:17.636 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:17.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:17.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:17.640+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:17.641 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:17.640+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:17.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:17.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:17.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:17.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:17.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:17.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:17.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:17.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:18.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:09:18.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:18.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:18.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:18.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:18.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:18.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:18.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:18.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:18.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:18.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:18.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:18.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:18.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:18.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:18.512+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:09:18.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:18.512+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:18.531 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:18.528+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:18.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:18.576+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:18.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:18.576+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:18.578 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:18.576+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:19.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:19.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:19.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:19.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:19.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:19.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:19.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:19.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:19.366 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:19.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:19.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:19.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:19.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:19.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:19.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:19.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:19.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:19.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:20.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:20.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:20.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:20.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:20.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:20.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:22.720 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:22.720+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:22.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:22.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:23.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:23.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:23.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:23.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:23.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:23.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:23.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:23.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:23.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:23.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:24.762 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:24.760+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:24.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:24.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:24.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:24.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:24.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:24.776+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:25.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:25.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:25.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:25.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:25.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:25.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:25.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:25.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:26.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:26.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:26.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:26.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:26.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:26.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:26.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:26.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:27.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:27.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:27.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:27.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:27.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:27.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:27.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:27.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:27.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:27.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:27.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:27.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:27.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:28.053 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:28.052+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:28.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:28.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:28.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:28.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:28.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:28.064+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:28.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:28.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:28.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:28.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:28.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:28.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:28.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:28.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:28.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:28.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:28.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:28.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:29.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:09:29.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:29.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:29.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:29.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:29.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:29.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:29.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:29.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:29.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:29.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:09:29.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:29.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:29.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:29.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:29.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:29.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:29.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:29.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:29.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:29.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:31.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:31.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:31.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:31.548+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:31.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:31.548+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:32.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:32.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:32.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:32.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:32.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:32.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:33.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:33.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:33.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:33.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:33.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:33.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:33.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:33.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:33.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:33.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:33.442 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:33.440+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:33.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:33.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:33.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:33.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:33.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:33.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:33.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:33.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:33.464+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:35.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:35.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:35.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:35.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:35.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.480+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:35.487 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:09:35.549 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.548+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:35.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.548+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:35.550 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:35.548+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:36.162 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:09:36.163 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:09:36.238 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2342,"stamp":"2023-10-14T01:09:35.585505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27958975320,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2177311,"num_read_kb":8461275,"num_write":2135316,"num_write_kb":8541665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80618,"ondisk_log_size":80618,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21290040,"kb_used_data":17079776,"kb_used_omap":1209,"kb_used_meta":4209030,"kb_avail":261825480,"statfs":{"total":289910292480,"available":268109291520,"internally_reserved":0,"allocated":17489690624,"data_stored":21799863015,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1238605,"internal_metadata":4310047155},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,4,8,7,10,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16977920,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4361,"num_read_kb":17388,"num_write":4176,"num_write_kb":16704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004470"},"pg_stats":[{"pgid":"2.7","version":"140'286369","reported_seq":562560,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.196210+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:09:34.196210+0000","last_peered":"2023-10-14T01:09:34.196210+0000","last_clean":"2023-10-14T01:09:34.196210+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:09:34.196210+0000","last_undegraded":"2023-10-14T01:09:34.196210+0000","last_fullsized":"2023-10-14T01:09:34.196210+0000","mapping_epoch":13,"log_start":"137'276302","ondisk_log_start":"137'276302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3535946166,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":277686,"num_read_kb":1073555,"num_write":269609,"num_write_kb":1078194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"140'286402","reported_seq":563118,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.136557+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:09:34.136557+0000","last_peered":"2023-10-14T01:09:34.136557+0000","last_clean":"2023-10-14T01:09:34.136557+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:09:34.136557+0000","last_undegraded":"2023-10-14T01:09:34.136557+0000","last_fullsized":"2023-10-14T01:09:34.136557+0000","mapping_epoch":13,"log_start":"137'276309","ondisk_log_start":"137'276309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3529056624,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":279812,"num_read_kb":1072073,"num_write":269661,"num_write_kb":1078271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"140'258992","reported_seq":515755,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.127008+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:09:34.127008+0000","last_peered":"2023-10-14T01:09:34.127008+0000","last_clean":"2023-10-14T01:09:34.127008+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:09:34.127008+0000","last_undegraded":"2023-10-14T01:09:34.127008+0000","last_fullsized":"2023-10-14T01:09:34.127008+0000","mapping_epoch":13,"log_start":"137'248905","ondisk_log_start":"137'248905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3376218514,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":257195,"num_read_kb":1013136,"num_write":258374,"num_write_kb":1033466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"140'253023","reported_seq":506771,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.104501+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:09:34.104501+0000","last_peered":"2023-10-14T01:09:34.104501+0000","last_clean":"2023-10-14T01:09:34.104501+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:09:34.104501+0000","last_undegraded":"2023-10-14T01:09:34.104501+0000","last_fullsized":"2023-10-14T01:09:34.104501+0000","mapping_epoch":13,"log_start":"137'243004","ondisk_log_start":"137'243004","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3304870019,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":254221,"num_read_kb":999489,"num_write":252418,"num_write_kb":1009594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"140'285489","reported_seq":571942,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:31.799992+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:09:31.799992+0000","last_peered":"2023-10-14T01:09:31.799992+0000","last_clean":"2023-10-14T01:09:31.799992+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:09:31.799992+0000","last_undegraded":"2023-10-14T01:09:31.799992+0000","last_fullsized":"2023-10-14T01:09:31.799992+0000","mapping_epoch":13,"log_start":"137'275407","ondisk_log_start":"137'275407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":943,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 457s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3716432191,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":287028,"num_read_kb":1122674,"num_write":283378,"num_write_kb":1133365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"140'275501","reported_seq":546839,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:34.406553+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:09:34.406553+0000","last_peered":"2023-10-14T01:09:34.406553+0000","last_clean":"2023-10-14T01:09:34.406553+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:09:34.406553+0000","last_undegraded":"2023-10-14T01:09:34.406553+0000","last_fullsized":"2023-10-14T01:09:34.406553+0000","mapping_epoch":13,"log_start":"137'265409","ondisk_log_start":"137'265409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1207,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 454s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3486871821,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":272585,"num_read_kb":1054000,"num_write":266972,"num_write_kb":1067674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"140'279613","reported_seq":557806,"reported_epoch":140,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:34.393605+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:09:34.393605+0000","last_peered":"2023-10-14T01:09:34.393605+0000","last_clean":"2023-10-14T01:09:34.393605+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:09:34.393605+0000","last_undegraded":"2023-10-14T01:09:34.393605+0000","last_fullsized":"2023-10-14T01:09:34.393605+0000","mapping_epoch":13,"log_start":"137'269611","ondisk_log_start":"137'269611","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1150,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 458s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3535393017,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":279675,"num_read_kb":1076188,"num_write":269348,"num_write_kb":1077118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"140'274156","reported_seq":542216,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.128865+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:09:34.128865+0000","last_peered":"2023-10-14T01:09:34.128865+0000","last_clean":"2023-10-14T01:09:34.128865+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:09:34.128865+0000","last_undegraded":"2023-10-14T01:09:34.128865+0000","last_fullsized":"2023-10-14T01:09:34.128865+0000","mapping_epoch":13,"log_start":"137'264107","ondisk_log_start":"137'264107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3473596600,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":268983,"num_read_kb":1050051,"num_write":265387,"num_write_kb":1061435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":346,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T01:08:04.476910+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:08:04.476910+0000","last_peered":"2023-10-14T01:08:04.476910+0000","last_clean":"2023-10-14T01:08:04.476910+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:08:04.476910+0000","last_undegraded":"2023-10-14T01:08:04.476910+0000","last_fullsized":"2023-10-14T01:08:04.476910+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27958384952,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2177185,"num_read_kb":8461166,"num_write":2135147,"num_write_kb":8539117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17486741504,"data_stored":21798220400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80491,"ondisk_log_size":80491,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739249,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4974760,"kb_used_data":3207008,"kb_used_omap":212,"kb_used_meta":1767531,"kb_avail":89397080,"statfs":{"total":96636764160,"available":91542609920,"internally_reserved":0,"allocated":3283976192,"data_stored":4093157755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1809951988},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.94899999999999995,"15min":0.84599999999999997},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.14699999999999999},"max":{"1min":2.673,"5min":7.492,"15min":9.4209999999999994},"last":1.6040000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82599999999999996,"15min":0.81499999999999995},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":4.3579999999999997,"5min":7.782,"15min":10.847},"last":0.78600000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:09:09 2023","interfaces":[{"interface":"back","average":{"1min":2.109,"5min":1.532,"15min":1.23},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":7.9589999999999996,"5min":9.657,"15min":14.103999999999999},"last":1.339},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.198,"15min":1.208},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.19},"max":{"1min":8.9420000000000002,"5min":8.9420000000000002,"15min":16.736999999999998},"last":1.7210000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739237,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9795828,"kb_used_data":8540844,"kb_used_omap":623,"kb_used_meta":1254352,"kb_avail":84576012,"statfs":{"total":96636764160,"available":86605836288,"internally_reserved":0,"allocated":8745824256,"data_stored":10901305877,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1284456937},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:26 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.51900000000000002,"15min":0.76900000000000002},"min":{"1min":0.248,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":12.369999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.65800000000000003,"15min":0.80700000000000005},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":4.601,"5min":5.8040000000000003,"15min":12.407},"last":0.47899999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:08:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":1.083,"15min":1.1020000000000001},"min":{"1min":0.246,"5min":0.222,"15min":0.22},"max":{"1min":3.5739999999999998,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.438},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.2769999999999999,"15min":1.0880000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":3.2469999999999999,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739247,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6519452,"kb_used_data":5331924,"kb_used_omap":373,"kb_used_meta":1187146,"kb_avail":87852388,"statfs":{"total":96636764160,"available":89960845312,"internally_reserved":0,"allocated":5459890176,"data_stored":6805399383,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1215638230},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,4,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.746,"15min":0.97399999999999998},"min":{"1min":0.216,"5min":0.182,"15min":0.17699999999999999},"max":{"1min":5.109,"5min":6.7290000000000001,"15min":12.855},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.627,"15min":0.87},"min":{"1min":0.221,"5min":0.17499999999999999,"15min":0.17399999999999999},"max":{"1min":3.4750000000000001,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:09:19 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.343,"15min":1.4219999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":3.9670000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71099999999999997,"15min":0.76100000000000001},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":3.4100000000000001,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.314}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5458952192,"data_stored":6804719226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8744566784,"data_stored":10900306232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3283222528,"data_stored":4093194942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:09:36.240 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.240 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.241 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.241 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.242 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.242 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.242 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.243 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:36.243 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:09:37.138 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:37.132+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:37.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:37.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:37.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:37.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:37.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:37.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:37.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:37.184+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:39.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:39.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:39.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:39.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:39.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:39.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:39.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:39.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:39.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:41.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:41.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:41.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:41.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:41.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:41.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:41.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:41.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:41.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:41.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:44.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:44.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:44.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:44.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:44.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:44.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:44.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:44.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:44.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:44.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:44.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:44.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:44.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:44.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:44.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:44.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:44.964 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:44.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:45.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:45.520+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:45.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:45.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:45.584 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:45.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:47.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:47.580+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:47.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:47.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:47.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:47.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:47.612 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:47.608+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:49.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:49.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:09:49.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:49.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:49.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:49.036+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:49.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:49.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:49.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:49.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:49.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:49.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:49.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:49.088+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:49.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:49.088+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:51.255 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:51.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:51.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:51.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:51.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:51.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:51.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:51.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:51.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:51.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:51.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:51.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:52.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:52.788+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:52.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:52.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:52.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:52.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:52.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:52.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:52.818 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:52.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:53.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:53.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:53.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:53.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:53.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:53.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:53.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:53.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:09:53.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:53.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:53.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:53.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.676+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:53.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:53.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:53.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:53.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:53.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:53.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:53.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:53.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:53.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:53.788 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.784+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:53.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:53.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:54.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:54.800+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:54.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:54.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:54.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:54.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:54.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:54.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:55.010 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.008+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:55.027 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.024+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:55.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:55.041 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:55.041 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:55.041 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:55.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:09:55.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.340+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:09:55.358 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.356+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:09:55.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:55.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:55.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:09:55.395 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.392+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:09:55.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:55.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:09:55.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:55.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:55.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:55.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:55.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:55.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:55.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:55.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:55.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:55.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:55.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:09:55.435 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:09:55.432+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:09:56.244 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:09:56.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:56.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:56.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:56.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:56.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:56.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:56.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:56.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:56.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:56.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:09:57.143 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:09:57.144 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:09:57.177 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2352,"stamp":"2023-10-14T01:09:55.597636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27993480024,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2186127,"num_read_kb":8496371,"num_write":2143747,"num_write_kb":8575389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80401,"ondisk_log_size":80401,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21643032,"kb_used_data":17147004,"kb_used_omap":1209,"kb_used_meta":4494790,"kb_avail":261472488,"statfs":{"total":289910292480,"available":267747827712,"internally_reserved":0,"allocated":17558532096,"data_stored":21868681276,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1238605,"internal_metadata":4602665395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,6,5,3,12,10,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24629248,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6364,"num_read_kb":25324,"num_write":6015,"num_write_kb":24060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006872"},"pg_stats":[{"pgid":"2.7","version":"141'287473","reported_seq":564821,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:54.528776+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:09:54.528776+0000","last_peered":"2023-10-14T01:09:54.528776+0000","last_clean":"2023-10-14T01:09:54.528776+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:09:54.528776+0000","last_undegraded":"2023-10-14T01:09:54.528776+0000","last_fullsized":"2023-10-14T01:09:54.528776+0000","mapping_epoch":13,"log_start":"137'277402","ondisk_log_start":"137'277402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3540361654,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":278825,"num_read_kb":1078087,"num_write":270680,"num_write_kb":1082478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'287499","reported_seq":565349,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:54.545466+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:09:54.545466+0000","last_peered":"2023-10-14T01:09:54.545466+0000","last_clean":"2023-10-14T01:09:54.545466+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:09:54.545466+0000","last_undegraded":"2023-10-14T01:09:54.545466+0000","last_fullsized":"2023-10-14T01:09:54.545466+0000","mapping_epoch":13,"log_start":"137'277409","ondisk_log_start":"137'277409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3533386096,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":280928,"num_read_kb":1076505,"num_write":270723,"num_write_kb":1082519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'260025","reported_seq":517876,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:54.524201+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:09:54.524201+0000","last_peered":"2023-10-14T01:09:54.524201+0000","last_clean":"2023-10-14T01:09:54.524201+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:09:54.524201+0000","last_undegraded":"2023-10-14T01:09:54.524201+0000","last_fullsized":"2023-10-14T01:09:54.524201+0000","mapping_epoch":13,"log_start":"137'250005","ondisk_log_start":"137'250005","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3380416914,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":258282,"num_read_kb":1017476,"num_write":259407,"num_write_kb":1037598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'254024","reported_seq":508822,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:54.510913+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:09:54.510913+0000","last_peered":"2023-10-14T01:09:54.510913+0000","last_clean":"2023-10-14T01:09:54.510913+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:09:54.510913+0000","last_undegraded":"2023-10-14T01:09:54.510913+0000","last_fullsized":"2023-10-14T01:09:54.510913+0000","mapping_epoch":13,"log_start":"137'244004","ondisk_log_start":"137'244004","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3308961923,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":255268,"num_read_kb":1003653,"num_write":253419,"num_write_kb":1013598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"141'286610","reported_seq":574241,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:51.929446+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:09:51.929446+0000","last_peered":"2023-10-14T01:09:51.929446+0000","last_clean":"2023-10-14T01:09:51.929446+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:09:51.929446+0000","last_undegraded":"2023-10-14T01:09:51.929446+0000","last_fullsized":"2023-10-14T01:09:51.929446+0000","mapping_epoch":13,"log_start":"138'276607","ondisk_log_start":"138'276607","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":979,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 477s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3720937791,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":288192,"num_read_kb":1127310,"num_write":284474,"num_write_kb":1137749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"141'276637","reported_seq":549107,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:54.643806+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:09:54.643806+0000","last_peered":"2023-10-14T01:09:54.643806+0000","last_clean":"2023-10-14T01:09:54.643806+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:09:54.643806+0000","last_undegraded":"2023-10-14T01:09:54.643806+0000","last_fullsized":"2023-10-14T01:09:54.643806+0000","mapping_epoch":13,"log_start":"137'266609","ondisk_log_start":"137'266609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1247,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 474s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3491242253,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":273697,"num_read_kb":1058420,"num_write":268040,"num_write_kb":1071946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"141'280731","reported_seq":560041,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:09:54.646931+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:09:54.646931+0000","last_peered":"2023-10-14T01:09:54.646931+0000","last_clean":"2023-10-14T01:09:54.646931+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:09:54.646931+0000","last_undegraded":"2023-10-14T01:09:54.646931+0000","last_fullsized":"2023-10-14T01:09:54.646931+0000","mapping_epoch":13,"log_start":"138'270711","ondisk_log_start":"138'270711","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1186,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 478s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3539665145,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":280766,"num_read_kb":1080532,"num_write":270393,"num_write_kb":1081298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"141'275229","reported_seq":544356,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:54.488386+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:09:54.488386+0000","last_peered":"2023-10-14T01:09:54.488386+0000","last_clean":"2023-10-14T01:09:54.488386+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:09:54.488386+0000","last_undegraded":"2023-10-14T01:09:54.488386+0000","last_fullsized":"2023-10-14T01:09:54.488386+0000","mapping_epoch":13,"log_start":"137'265207","ondisk_log_start":"137'265207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3477917880,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":270043,"num_read_kb":1054279,"num_write":266442,"num_write_kb":1065655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":348,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.511725+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:09:34.511725+0000","last_peered":"2023-10-14T01:09:34.511725+0000","last_clean":"2023-10-14T01:09:34.511725+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:09:34.511725+0000","last_undegraded":"2023-10-14T01:09:34.511725+0000","last_fullsized":"2023-10-14T01:09:34.511725+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":27992889656,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2186001,"num_read_kb":8496262,"num_write":2143578,"num_write_kb":8572841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17555668992,"data_stored":21867147888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80274,"ondisk_log_size":80274,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739253,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5066772,"kb_used_data":3220040,"kb_used_omap":212,"kb_used_meta":1846507,"kb_avail":89305068,"statfs":{"total":96636764160,"available":91448389632,"internally_reserved":0,"allocated":3297320960,"data_stored":4106494786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1890823412},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:09 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.94899999999999995,"15min":0.84599999999999997},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.14699999999999999},"max":{"1min":2.673,"5min":7.492,"15min":9.4209999999999994},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.82599999999999996,"15min":0.81499999999999995},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.191},"max":{"1min":4.3579999999999997,"5min":7.782,"15min":10.847},"last":1.0960000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:09:09 2023","interfaces":[{"interface":"back","average":{"1min":2.109,"5min":1.532,"15min":1.23},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":7.9589999999999996,"5min":9.657,"15min":14.103999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.198,"15min":1.208},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.19},"max":{"1min":8.9420000000000002,"5min":8.9420000000000002,"15min":16.736999999999998},"last":1.6890000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739241,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9959412,"kb_used_data":8573996,"kb_used_omap":623,"kb_used_meta":1384784,"kb_avail":84412428,"statfs":{"total":96636764160,"available":86438326272,"internally_reserved":0,"allocated":8779771904,"data_stored":10935245788,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1418019305},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,2,3,8,6,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:26 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.51900000000000002,"15min":0.76900000000000002},"min":{"1min":0.248,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":12.369999999999999},"last":0.215},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.65800000000000003,"15min":0.80700000000000005},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":4.601,"5min":5.8040000000000003,"15min":12.407},"last":0.255}]},{"osd":2,"last update":"Sat Oct 14 01:09:41 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.22,"15min":1.1619999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.22},"max":{"1min":8.8149999999999995,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4350000000000001,"15min":1.117},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":5.3819999999999997,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.47199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739251,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6616848,"kb_used_data":5352968,"kb_used_omap":373,"kb_used_meta":1263498,"kb_avail":87754992,"statfs":{"total":96636764160,"available":89861111808,"internally_reserved":0,"allocated":5481439232,"data_stored":6826940702,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1293822678},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.746,"15min":0.97399999999999998},"min":{"1min":0.216,"5min":0.182,"15min":0.17699999999999999},"max":{"1min":5.109,"5min":6.7290000000000001,"15min":12.855},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.627,"15min":0.87},"min":{"1min":0.221,"5min":0.17499999999999999,"15min":0.17399999999999999},"max":{"1min":3.4750000000000001,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.22900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:09:19 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.343,"15min":1.4219999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":3.9670000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.248},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71099999999999997,"15min":0.76100000000000001},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":3.4100000000000001,"5min":5.2329999999999997,"15min":15.451000000000001},"last":0.998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5480275968,"data_stored":6826043002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8779218944,"data_stored":10934958392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3296174080,"data_stored":4106146494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:09:57.179 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.179 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.179 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.180 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.180 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.180 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.181 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.181 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:09:57.181 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:09:58.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:58.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:09:58.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:58.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:09:58.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:58.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:09:58.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:58.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:09:58.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:58.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:09:58.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:58.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:09:58.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:09:58.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:00.273 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:00.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:00.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:00.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:00.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:00.279+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:00.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:00.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:00.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:00.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:00.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:00.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:00.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:00.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:01.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:01.537 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:01.537 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:01.538 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.535+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:01.811 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.807+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:01.835 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:01.848 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:01.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:01.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:01.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:02.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:02.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:02.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:02.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:02.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:02.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:02.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:02.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:02.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:02.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:02.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:02.407+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:05.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:05.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:05.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:05.979+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:05.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:05.987+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:06.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:06.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:06.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:06.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:06.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:06.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:06.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:06.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:06.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:06.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:07.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:07.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:07.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:07.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:07.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:07.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:07.256 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:07.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:07.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:07.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:07.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:07.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:07.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:09.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:09.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:09.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:09.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:09.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.187+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:09.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:10:09.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:10:09.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:09.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:09.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:09.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:09.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:09.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:09.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:09.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:09.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:09.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:10.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:10.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:11.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:11.235+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:11.260 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:11.255+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:12.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:12.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:12.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:12.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:12.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:12.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:12.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:12.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:12.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:12.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:12.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:12.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:12.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:12.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:12.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:12.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:12.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:12.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:12.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:12.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:12.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:12.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:13.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.031+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:13.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:13.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:13.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:13.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:13.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:13.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:13.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:13.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:13.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:13.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:13.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:15.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:10:15.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:10:15.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.251+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:15.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:15.291 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:15.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:15.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:15.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:15.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:15.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:15.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:16.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:16.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:16.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:16.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:16.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:16.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:10:16.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:16.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:16.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:16.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:16.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:16.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:16.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:16.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:16.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:16.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:16.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:16.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:16.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:16.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:17.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:17.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:17.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:17.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:17.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:17.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:17.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:17.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:17.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:17.183 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:17.998 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:10:17.999 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:10:18.042 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2363,"stamp":"2023-10-14T01:10:17.605077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28029004632,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2195213,"num_read_kb":8532547,"num_write":2152428,"num_write_kb":8610113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80375,"ondisk_log_size":80375,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22020020,"kb_used_data":17218072,"kb_used_omap":1209,"kb_used_meta":4800710,"kb_avail":261095500,"statfs":{"total":289910292480,"available":267361792000,"internally_reserved":0,"allocated":17631305728,"data_stored":21941454908,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1238605,"internal_metadata":4915927475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,6,6,10,11,16,18,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18247680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4615,"num_read_kb":18380,"num_write":4455,"num_write_kb":17820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005770"},"pg_stats":[{"pgid":"2.7","version":"141'288564","reported_seq":567031,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:15.018366+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:10:15.018366+0000","last_peered":"2023-10-14T01:10:15.018366+0000","last_clean":"2023-10-14T01:10:15.018366+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:10:15.018366+0000","last_undegraded":"2023-10-14T01:10:15.018366+0000","last_fullsized":"2023-10-14T01:10:15.018366+0000","mapping_epoch":13,"log_start":"138'278502","ondisk_log_start":"138'278502","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3544613302,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":279932,"num_read_kb":1082495,"num_write":271711,"num_write_kb":1086602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'288619","reported_seq":567591,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:14.996561+0000","last_change":"2023-10-13T23:57:08.679068+0000","last_active":"2023-10-14T01:10:14.996561+0000","last_peered":"2023-10-14T01:10:14.996561+0000","last_clean":"2023-10-14T01:10:14.996561+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:10:14.996561+0000","last_undegraded":"2023-10-14T01:10:14.996561+0000","last_fullsized":"2023-10-14T01:10:14.996561+0000","mapping_epoch":13,"log_start":"138'278609","ondisk_log_start":"138'278609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3537699184,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":282032,"num_read_kb":1080905,"num_write":271783,"num_write_kb":1086759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'261015","reported_seq":519916,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:15.005993+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:10:15.005993+0000","last_peered":"2023-10-14T01:10:15.005993+0000","last_clean":"2023-10-14T01:10:15.005993+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:10:15.005993+0000","last_undegraded":"2023-10-14T01:10:15.005993+0000","last_fullsized":"2023-10-14T01:10:15.005993+0000","mapping_epoch":13,"log_start":"137'251005","ondisk_log_start":"137'251005","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3384480146,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":259331,"num_read_kb":1021664,"num_write":260397,"num_write_kb":1041558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'255016","reported_seq":510843,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:14.982872+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:10:14.982872+0000","last_peered":"2023-10-14T01:10:14.982872+0000","last_clean":"2023-10-14T01:10:14.982872+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:10:14.982872+0000","last_undegraded":"2023-10-14T01:10:14.982872+0000","last_fullsized":"2023-10-14T01:10:14.982872+0000","mapping_epoch":13,"log_start":"137'245004","ondisk_log_start":"137'245004","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3313008771,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":256296,"num_read_kb":1007745,"num_write":254411,"num_write_kb":1017566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"141'288024","reported_seq":577140,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:17.093263+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:10:17.093263+0000","last_peered":"2023-10-14T01:10:17.093263+0000","last_clean":"2023-10-14T01:10:17.093263+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:10:17.093263+0000","last_undegraded":"2023-10-14T01:10:17.093263+0000","last_fullsized":"2023-10-14T01:10:17.093263+0000","mapping_epoch":13,"log_start":"138'278007","ondisk_log_start":"138'278007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1001,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 502s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3726684479,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":289663,"num_read_kb":1133154,"num_write":285871,"num_write_kb":1143337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"141'277749","reported_seq":551330,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:14.840985+0000","last_change":"2023-10-14T01:02:00.046121+0000","last_active":"2023-10-14T01:10:14.840985+0000","last_peered":"2023-10-14T01:10:14.840985+0000","last_clean":"2023-10-14T01:10:14.840985+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:10:14.840985+0000","last_undegraded":"2023-10-14T01:10:14.840985+0000","last_fullsized":"2023-10-14T01:10:14.840985+0000","mapping_epoch":13,"log_start":"138'267709","ondisk_log_start":"138'267709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1272,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 494s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3495579917,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":274784,"num_read_kb":1062752,"num_write":269106,"num_write_kb":1076210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"141'281885","reported_seq":562304,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:14.853962+0000","last_change":"2023-10-14T01:01:56.094331+0000","last_active":"2023-10-14T01:10:14.853962+0000","last_peered":"2023-10-14T01:10:14.853962+0000","last_clean":"2023-10-14T01:10:14.853962+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:10:14.853962+0000","last_undegraded":"2023-10-14T01:10:14.853962+0000","last_fullsized":"2023-10-14T01:10:14.853962+0000","mapping_epoch":13,"log_start":"138'271811","ondisk_log_start":"138'271811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1258,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 498s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3544047865,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":281846,"num_read_kb":1084828,"num_write":271467,"num_write_kb":1085594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"141'276330","reported_seq":546623,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:15.009699+0000","last_change":"2023-10-13T23:57:08.678120+0000","last_active":"2023-10-14T01:10:15.009699+0000","last_peered":"2023-10-14T01:10:15.009699+0000","last_clean":"2023-10-14T01:10:15.009699+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:10:15.009699+0000","last_undegraded":"2023-10-14T01:10:15.009699+0000","last_fullsized":"2023-10-14T01:10:15.009699+0000","mapping_epoch":13,"log_start":"138'266307","ondisk_log_start":"138'266307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3482300600,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":271203,"num_read_kb":1058895,"num_write":267513,"num_write_kb":1069939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":348,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.511725+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:09:34.511725+0000","last_peered":"2023-10-14T01:09:34.511725+0000","last_clean":"2023-10-14T01:09:34.511725+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:09:34.511725+0000","last_undegraded":"2023-10-14T01:09:34.511725+0000","last_fullsized":"2023-10-14T01:09:34.511725+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28028414264,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2195087,"num_read_kb":8532438,"num_write":2152259,"num_write_kb":8607565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17629433856,"data_stored":21940912752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80248,"ondisk_log_size":80248,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739257,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5133612,"kb_used_data":3232740,"kb_used_omap":212,"kb_used_meta":1900651,"kb_avail":89238228,"statfs":{"total":96636764160,"available":91379945472,"internally_reserved":0,"allocated":3310325760,"data_stored":4119499586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1946266868},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,4,2,7,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.14699999999999999},"max":{"1min":1.6040000000000001,"5min":7.492,"15min":9.4209999999999994},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.81000000000000005,"15min":0.83699999999999997},"min":{"1min":0.222,"5min":0.217,"15min":0.191},"max":{"1min":3.5499999999999998,"5min":7.782,"15min":10.847},"last":0.22900000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.1459999999999999,"15min":1.238},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.191},"max":{"1min":2.875,"5min":7.9589999999999996,"15min":14.103999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1499999999999999,"15min":1.1439999999999999},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.19},"max":{"1min":4.8650000000000002,"5min":8.9420000000000002,"15min":12.928000000000001},"last":1.6279999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739245,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10135556,"kb_used_data":8606776,"kb_used_omap":623,"kb_used_meta":1528144,"kb_avail":84236284,"statfs":{"total":96636764160,"available":86257954816,"internally_reserved":0,"allocated":8813338624,"data_stored":10968812508,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1564819945},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,1,5,6,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:26 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.51900000000000002,"15min":0.76900000000000002},"min":{"1min":0.248,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":12.369999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.65800000000000003,"15min":0.80700000000000005},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":4.601,"5min":5.8040000000000003,"15min":12.407},"last":0.432}]},{"osd":2,"last update":"Sat Oct 14 01:09:41 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.22,"15min":1.1619999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.22},"max":{"1min":8.8149999999999995,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4350000000000001,"15min":1.117},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":5.3819999999999997,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359739256,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6750852,"kb_used_data":5378556,"kb_used_omap":373,"kb_used_meta":1371914,"kb_avail":87620988,"statfs":{"total":96636764160,"available":89723891712,"internally_reserved":0,"allocated":5507641344,"data_stored":6853142814,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1404840662},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,1,3,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:19 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.746,"15min":0.97399999999999998},"min":{"1min":0.216,"5min":0.182,"15min":0.17699999999999999},"max":{"1min":5.109,"5min":6.7290000000000001,"15min":12.855},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.627,"15min":0.87},"min":{"1min":0.221,"5min":0.17499999999999999,"15min":0.17399999999999999},"max":{"1min":3.4750000000000001,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.309}]},{"osd":2,"last update":"Sat Oct 14 01:09:19 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.343,"15min":1.4219999999999999},"min":{"1min":0.23599999999999999,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":3.9670000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":4.3399999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.71099999999999997,"15min":0.76100000000000001},"min":{"1min":0.255,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":3.4100000000000001,"5min":5.2329999999999997,"15min":15.451000000000001},"last":2.3210000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5506457600,"data_stored":6852224634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8813719552,"data_stored":10969459000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3309256704,"data_stored":4119229118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:10:18.044 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.044 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.044 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.045 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.045 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.046 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.046 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.047 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:18.047 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:10:21.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:21.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:21.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:21.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:21.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:21.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:21.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:21.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:21.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:22.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:22.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:22.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:22.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:22.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:22.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:22.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:22.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:22.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:22.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:23.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:10:23.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:23.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:23.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:23.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:23.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:23.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:23.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:23.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:23.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:24.198 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:24.198 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.195+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:24.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.199+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:24.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.203+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:24.207 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.203+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:24.207 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.203+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:24.208 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.203+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:24.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.695+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:24.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.695+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:24.713 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.707+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:24.713 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.711+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:24.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:24.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:24.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:24.719+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:25.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:10:25.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:10:25.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:25.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:25.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:25.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:25.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:25.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:25.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:25.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:25.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:26.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:26.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:10:26.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:26.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:26.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:26.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:26.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:26.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:26.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:26.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:26.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:26.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:26.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:26.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:27.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:27.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:27.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:27.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:27.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:27.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:27.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:27.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:27.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:27.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:29.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:29.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:29.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:29.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:29.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:29.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:29.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:29.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:29.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:29.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:29.970 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:29.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:29.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:29.975+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:29.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:29.987+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:29.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:29.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:29.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:29.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:29.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:29.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:32.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:32.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:32.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:32.052 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.047+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:32.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:32.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:32.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:32.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.399+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:32.417 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.415+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:32.425 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.423+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:32.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:32.458 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:32.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:32.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:32.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:32.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:32.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:32.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:32.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:32.487+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:32.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:32.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:32.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:32.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:32.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:32.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:34.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:34.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:34.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:34.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:34.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:34.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:34.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:34.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:34.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:34.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:34.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:34.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:34.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:34.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:35.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:35.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:35.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:35.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:35.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:35.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:35.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:35.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:35.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:35.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:35.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:35.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:35.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:35.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:36.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:36.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:36.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:36.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:36.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:36.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:36.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:36.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:37.477 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:37.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:37.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:37.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:38.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:38.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:38.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:38.048 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:38.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:38.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.387+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:38.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:38.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:38.784 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:10:38.785 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:10:38.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:38.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.791+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:38.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:38.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:38.837 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2373,"stamp":"2023-10-14T01:10:37.609175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28059999064,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2203188,"num_read_kb":8564367,"num_write":2159984,"num_write_kb":8640337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80566,"ondisk_log_size":80566,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22261572,"kb_used_data":17279204,"kb_used_omap":1209,"kb_used_meta":4981126,"kb_avail":260853948,"statfs":{"total":289910292480,"available":267114442752,"internally_reserved":0,"allocated":17693904896,"data_stored":22004054076,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1238605,"internal_metadata":5100673459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,11,6,5,11,20,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16351232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4235,"num_read_kb":16904,"num_write":3998,"num_write_kb":15992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001870"},"pg_stats":[{"pgid":"2.7","version":"141'289637","reported_seq":569110,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:35.479081+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:10:35.479081+0000","last_peered":"2023-10-14T01:10:35.479081+0000","last_clean":"2023-10-14T01:10:35.479081+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:10:35.479081+0000","last_undegraded":"2023-10-14T01:10:35.479081+0000","last_fullsized":"2023-10-14T01:10:35.479081+0000","mapping_epoch":13,"log_start":"138'279602","ondisk_log_start":"138'279602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3548721590,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":280918,"num_read_kb":1086427,"num_write":272717,"num_write_kb":1090626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"141'289672","reported_seq":569641,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:35.484342+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:10:35.484342+0000","last_peered":"2023-10-14T01:10:35.484342+0000","last_clean":"2023-10-14T01:10:35.484342+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:10:35.484342+0000","last_undegraded":"2023-10-14T01:10:35.484342+0000","last_fullsized":"2023-10-14T01:10:35.484342+0000","mapping_epoch":13,"log_start":"138'279609","ondisk_log_start":"138'279609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":2,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 7s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3541741936,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":283009,"num_read_kb":1084805,"num_write":272768,"num_write_kb":1090699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"141'261921","reported_seq":521806,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:35.455826+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:10:35.455826+0000","last_peered":"2023-10-14T01:10:35.455826+0000","last_clean":"2023-10-14T01:10:35.455826+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:10:35.455826+0000","last_undegraded":"2023-10-14T01:10:35.455826+0000","last_fullsized":"2023-10-14T01:10:35.455826+0000","mapping_epoch":13,"log_start":"138'251905","ondisk_log_start":"138'251905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3388166546,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":260315,"num_read_kb":1025600,"num_write":261303,"num_write_kb":1045182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"141'255843","reported_seq":512639,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:35.467148+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:10:35.467148+0000","last_peered":"2023-10-14T01:10:35.467148+0000","last_clean":"2023-10-14T01:10:35.467148+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:10:35.467148+0000","last_undegraded":"2023-10-14T01:10:35.467148+0000","last_fullsized":"2023-10-14T01:10:35.467148+0000","mapping_epoch":13,"log_start":"138'245804","ondisk_log_start":"138'245804","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3316396163,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":257267,"num_read_kb":1011625,"num_write":255238,"num_write_kb":1020874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"141'289086","reported_seq":579221,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:37.170627+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:10:37.170627+0000","last_peered":"2023-10-14T01:10:37.170627+0000","last_clean":"2023-10-14T01:10:37.170627+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:10:37.170627+0000","last_undegraded":"2023-10-14T01:10:37.170627+0000","last_fullsized":"2023-10-14T01:10:37.170627+0000","mapping_epoch":13,"log_start":"138'279007","ondisk_log_start":"138'279007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1089,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 522s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3730809151,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":290666,"num_read_kb":1137154,"num_write":286884,"num_write_kb":1147389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"141'278697","reported_seq":553300,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:34.946398+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:10:34.946398+0000","last_peered":"2023-10-14T01:10:34.946398+0000","last_clean":"2023-10-14T01:10:34.946398+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:10:34.946398+0000","last_undegraded":"2023-10-14T01:10:34.946398+0000","last_fullsized":"2023-10-14T01:10:34.946398+0000","mapping_epoch":13,"log_start":"138'268609","ondisk_log_start":"138'268609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3499385101,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":275783,"num_read_kb":1066736,"num_write":270027,"num_write_kb":1079894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"141'282897","reported_seq":564344,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:10:34.946881+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:10:34.946881+0000","last_peered":"2023-10-14T01:10:34.946881+0000","last_clean":"2023-10-14T01:10:34.946881+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:10:34.946881+0000","last_undegraded":"2023-10-14T01:10:34.946881+0000","last_fullsized":"2023-10-14T01:10:34.946881+0000","mapping_epoch":13,"log_start":"138'272811","ondisk_log_start":"138'272811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3547975929,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":282843,"num_read_kb":1088792,"num_write":272426,"num_write_kb":1089430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"141'277340","reported_seq":548711,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:35.425859+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:10:35.425859+0000","last_peered":"2023-10-14T01:10:35.425859+0000","last_clean":"2023-10-14T01:10:35.425859+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:10:35.425859+0000","last_undegraded":"2023-10-14T01:10:35.425859+0000","last_fullsized":"2023-10-14T01:10:35.425859+0000","mapping_epoch":13,"log_start":"138'267307","ondisk_log_start":"138'267307","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":76,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3486212280,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":272261,"num_read_kb":1063119,"num_write":268452,"num_write_kb":1073695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":348,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:34.511725+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:09:34.511725+0000","last_peered":"2023-10-14T01:09:34.511725+0000","last_clean":"2023-10-14T01:09:34.511725+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:09:34.511725+0000","last_undegraded":"2023-10-14T01:09:34.511725+0000","last_fullsized":"2023-10-14T01:09:34.511725+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28059408696,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2203062,"num_read_kb":8564258,"num_write":2159815,"num_write_kb":8637789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17691131904,"data_stored":22002610800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80439,"ondisk_log_size":80439,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5172500,"kb_used_data":3243852,"kb_used_omap":212,"kb_used_meta":1928427,"kb_avail":89199340,"statfs":{"total":96636764160,"available":91340124160,"internally_reserved":0,"allocated":3321704448,"data_stored":4130878274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":217868,"internal_metadata":1974709492},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.14699999999999999},"max":{"1min":1.6040000000000001,"5min":7.492,"15min":9.4209999999999994},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.81000000000000005,"15min":0.83699999999999997},"min":{"1min":0.222,"5min":0.217,"15min":0.191},"max":{"1min":3.5499999999999998,"5min":7.782,"15min":10.847},"last":0.29299999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.1459999999999999,"15min":1.238},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.191},"max":{"1min":2.875,"5min":7.9589999999999996,"15min":14.103999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1499999999999999,"15min":1.1439999999999999},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.19},"max":{"1min":4.8650000000000002,"5min":8.9420000000000002,"15min":12.928000000000001},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739249,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10283584,"kb_used_data":8638260,"kb_used_omap":623,"kb_used_meta":1644688,"kb_avail":84088256,"statfs":{"total":96636764160,"available":86106374144,"internally_reserved":0,"allocated":8845578240,"data_stored":11001052124,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1684161001},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,5,8,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:26 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.65600000000000003,"15min":0.75700000000000001},"min":{"1min":0.20699999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":10.552,"5min":10.552,"15min":12.369999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.245,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":10.52,"5min":10.52,"15min":12.407},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:09:41 2023","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.22,"15min":1.1619999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.22},"max":{"1min":8.8149999999999995,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4350000000000001,"15min":1.117},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":5.3819999999999997,"5min":39.412999999999997,"15min":39.412999999999997},"last":1.6419999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739260,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6805488,"kb_used_data":5397092,"kb_used_omap":373,"kb_used_meta":1408010,"kb_avail":87566352,"statfs":{"total":96636764160,"available":89667944448,"internally_reserved":0,"allocated":5526622208,"data_stored":6872123678,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1441802966},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,2,0,3,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.65000000000000002,"15min":0.93700000000000006},"min":{"1min":0.245,"5min":0.182,"15min":0.17699999999999999},"max":{"1min":0.97399999999999998,"5min":6.7290000000000001,"15min":12.855},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.315,"5min":0.56200000000000006,"15min":0.84199999999999997},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.65700000000000003,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:10:21 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.208,"15min":1.4059999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":8.0060000000000002,"5min":10.539999999999999,"15min":10.539999999999999},"last":4.7320000000000002},{"interface":"front","average":{"1min":1.9510000000000001,"5min":0.94699999999999995,"15min":0.80700000000000005},"min":{"1min":0.253,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":8.9710000000000001,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.501}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5525483520,"data_stored":6871250554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8844914688,"data_stored":11000654136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3320733696,"data_stored":4130706110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:10:38.838 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:38.838 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:38.839 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:38.839 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:38.840 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:38.840 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:38.840 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:10:38.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:38.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:41.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:41.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:41.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:41.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:41.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:41.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:42.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:42.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:42.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:42.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:42.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:42.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:43.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:43.307+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:43.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:43.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:43.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:43.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:43.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:43.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:43.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:43.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:43.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:43.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:45.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:45.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:45.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:45.059+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:45.312 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:45.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:45.312 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:45.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:45.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:45.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:45.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:45.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:45.343 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:45.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:45.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:45.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:45.964 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:45.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:45.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:45.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:45.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:45.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:46.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:46.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:46.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:46.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:46.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:46.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:47.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:47.671 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:47.679 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:47.698 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.695+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:47.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:10:47.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:47.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:47.736 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.731+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:47.744 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:47.743+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:49.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:49.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:49.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:49.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:49.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:49.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:49.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:49.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:49.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:49.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:49.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:49.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:10:49.949 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:49.946+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:50.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:49.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:50.036 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:50.026+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:50.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:50.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:50.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:50.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:50.056 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:50.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:50.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:50.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:50.290 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:50.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:51.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:51.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:51.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:51.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:51.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.350+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:51.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:51.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:51.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:51.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:10:51.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:51.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:51.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:51.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:51.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.906+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:51.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:51.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:51.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:53.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:52.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:10:53.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:52.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:53.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:52.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:53.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:53.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:53.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:53.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:53.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:53.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:53.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:53.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:53.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:53.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:53.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:53.618+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:54.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:54.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:54.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:54.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:54.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:54.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:55.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:55.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:55.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:55.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:55.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:55.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:10:55.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:55.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:56.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:56.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.582+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:56.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.582+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:56.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.582+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:56.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:56.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:56.719 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:56.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:56.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:56.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:56.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:56.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:56.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:10:56.798 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:10:56.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:10:56.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:10:56.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:10:56.863 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:10:56.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:10:56.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:56.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:56.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:56.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:56.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:56.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:57.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:57.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:57.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:57.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:57.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:57.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:57.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:10:57.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:57.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:10:57.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:57.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:58.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:58.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:10:58.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:58.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:10:58.841 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:58.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:58.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:10:58.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:10:58.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:10:59.645 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:10:59.650 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:10:59.729 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2385,"stamp":"2023-10-14T01:10:59.618616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28093074264,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2211307,"num_read_kb":8596739,"num_write":2168066,"num_write_kb":8672665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80402,"ondisk_log_size":80402,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22540792,"kb_used_data":17344160,"kb_used_omap":1211,"kb_used_meta":5195396,"kb_avail":260574728,"statfs":{"total":289910292480,"available":266828521472,"internally_reserved":0,"allocated":17760419840,"data_stored":22070545680,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1240285,"internal_metadata":5320086307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,2,6,11,13,21,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16990208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4198,"num_read_kb":16744,"num_write":4133,"num_write_kb":16532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001946"},"pg_stats":[{"pgid":"2.7","version":"142'290824","reported_seq":571353,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:10:56.023240+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:10:56.023240+0000","last_peered":"2023-10-14T01:10:56.023240+0000","last_clean":"2023-10-14T01:10:56.023240+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:10:56.023240+0000","last_undegraded":"2023-10-14T01:10:56.023240+0000","last_fullsized":"2023-10-14T01:10:56.023240+0000","mapping_epoch":13,"log_start":"138'280802","ondisk_log_start":"138'280802","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3553178038,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":281960,"num_read_kb":1090575,"num_write":273808,"num_write_kb":1094990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'290854","reported_seq":571892,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:56.021904+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:10:56.021904+0000","last_peered":"2023-10-14T01:10:56.021904+0000","last_clean":"2023-10-14T01:10:56.021904+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:10:56.021904+0000","last_undegraded":"2023-10-14T01:10:56.021904+0000","last_fullsized":"2023-10-14T01:10:56.021904+0000","mapping_epoch":13,"log_start":"138'280810","ondisk_log_start":"138'280810","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":30,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 27s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3546165616,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":284057,"num_read_kb":1088977,"num_write":273842,"num_write_kb":1094995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'262916","reported_seq":523817,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:10:55.990842+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:10:55.990842+0000","last_peered":"2023-10-14T01:10:55.990842+0000","last_clean":"2023-10-14T01:10:55.990842+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:10:55.990842+0000","last_undegraded":"2023-10-14T01:10:55.990842+0000","last_fullsized":"2023-10-14T01:10:55.990842+0000","mapping_epoch":13,"log_start":"138'252905","ondisk_log_start":"138'252905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3392221586,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":261329,"num_read_kb":1029652,"num_write":262298,"num_write_kb":1049162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'256778","reported_seq":514542,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:10:56.006133+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:10:56.006133+0000","last_peered":"2023-10-14T01:10:56.006133+0000","last_clean":"2023-10-14T01:10:56.006133+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:10:56.006133+0000","last_undegraded":"2023-10-14T01:10:56.006133+0000","last_fullsized":"2023-10-14T01:10:56.006133+0000","mapping_epoch":13,"log_start":"138'246704","ondisk_log_start":"138'246704","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3320230019,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":258232,"num_read_kb":1015473,"num_write":256173,"num_write_kb":1024614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"142'290116","reported_seq":581257,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:57.349811+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:10:57.349811+0000","last_peered":"2023-10-14T01:10:57.349811+0000","last_clean":"2023-10-14T01:10:57.349811+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:10:57.349811+0000","last_undegraded":"2023-10-14T01:10:57.349811+0000","last_fullsized":"2023-10-14T01:10:57.349811+0000","mapping_epoch":13,"log_start":"138'280108","ondisk_log_start":"138'280108","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1143,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 542s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3734901055,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":291658,"num_read_kb":1141102,"num_write":287881,"num_write_kb":1151377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"142'279716","reported_seq":555319,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:10:55.063700+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:10:55.063700+0000","last_peered":"2023-10-14T01:10:55.063700+0000","last_clean":"2023-10-14T01:10:55.063700+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:10:55.063700+0000","last_undegraded":"2023-10-14T01:10:55.063700+0000","last_fullsized":"2023-10-14T01:10:55.063700+0000","mapping_epoch":13,"log_start":"138'269709","ondisk_log_start":"138'269709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3503431949,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":276764,"num_read_kb":1070652,"num_write":271022,"num_write_kb":1083874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"142'283902","reported_seq":566397,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:10:55.052472+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:10:55.052472+0000","last_peered":"2023-10-14T01:10:55.052472+0000","last_clean":"2023-10-14T01:10:55.052472+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:10:55.052472+0000","last_undegraded":"2023-10-14T01:10:55.052472+0000","last_fullsized":"2023-10-14T01:10:55.052472+0000","mapping_epoch":13,"log_start":"138'273811","ondisk_log_start":"138'273811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3551998201,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":283872,"num_read_kb":1092900,"num_write":273405,"num_write_kb":1093346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"142'278425","reported_seq":550857,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:55.950679+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:10:55.950679+0000","last_peered":"2023-10-14T01:10:55.950679+0000","last_clean":"2023-10-14T01:10:55.950679+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:10:55.950679+0000","last_undegraded":"2023-10-14T01:10:55.950679+0000","last_fullsized":"2023-10-14T01:10:55.950679+0000","mapping_epoch":13,"log_start":"138'268407","ondisk_log_start":"138'268407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":118,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 34s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3490357432,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":273309,"num_read_kb":1067299,"num_write":269468,"num_write_kb":1077759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":349,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:44.334764+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:09:44.334764+0000","last_peered":"2023-10-14T01:09:44.334764+0000","last_clean":"2023-10-14T01:09:44.334764+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:09:44.334764+0000","last_undegraded":"2023-10-14T01:09:44.334764+0000","last_fullsized":"2023-10-14T01:09:44.334764+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28092483896,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2211181,"num_read_kb":8596630,"num_write":2167897,"num_write_kb":8670117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17757200384,"data_stored":22068679280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80275,"ondisk_log_size":80275,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5180216,"kb_used_data":3255664,"kb_used_omap":214,"kb_used_meta":1924329,"kb_avail":89191624,"statfs":{"total":96636764160,"available":91332222976,"internally_reserved":0,"allocated":3333799936,"data_stored":4142965982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1970513508},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.82699999999999996,"15min":0.83399999999999996},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.14699999999999999},"max":{"1min":1.6040000000000001,"5min":7.492,"15min":9.4209999999999994},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.81000000000000005,"15min":0.83699999999999997},"min":{"1min":0.222,"5min":0.217,"15min":0.191},"max":{"1min":3.5499999999999998,"5min":7.782,"15min":10.847},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:10:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.1459999999999999,"15min":1.238},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.191},"max":{"1min":2.875,"5min":7.9589999999999996,"15min":14.103999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1499999999999999,"15min":1.1439999999999999},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.19},"max":{"1min":4.8650000000000002,"5min":8.9420000000000002,"15min":12.928000000000001},"last":0.83999999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739253,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10440616,"kb_used_data":8671328,"kb_used_omap":623,"kb_used_meta":1768656,"kb_avail":83931224,"statfs":{"total":96636764160,"available":85945573376,"internally_reserved":0,"allocated":8879439872,"data_stored":11034905976,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1811104233},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,4,4,8,17,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:26 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.65600000000000003,"15min":0.75700000000000001},"min":{"1min":0.20699999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":10.552,"5min":10.552,"15min":12.369999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.245,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":10.52,"5min":10.52,"15min":12.407},"last":0.247}]},{"osd":2,"last update":"Sat Oct 14 01:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.3540000000000001,"15min":1.169},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":8.9469999999999992,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.423,"15min":1.1160000000000001},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":5.3170000000000002,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.441}]}]},{"osd":0,"up_from":8,"seq":34359739264,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6919960,"kb_used_data":5417168,"kb_used_omap":373,"kb_used_meta":1502410,"kb_avail":87451880,"statfs":{"total":96636764160,"available":89550725120,"internally_reserved":0,"allocated":5547180032,"data_stored":6892673722,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1538468566},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,5,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.65000000000000002,"15min":0.93700000000000006},"min":{"1min":0.245,"5min":0.182,"15min":0.17699999999999999},"max":{"1min":0.97399999999999998,"5min":6.7290000000000001,"15min":12.855},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.56200000000000006,"15min":0.84199999999999997},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.65700000000000003,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:10:21 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.208,"15min":1.4059999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":8.0060000000000002,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.307},{"interface":"front","average":{"1min":1.9510000000000001,"5min":0.94699999999999995,"15min":0.80700000000000005},"min":{"1min":0.253,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":8.9710000000000001,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":3.9249999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5546057728,"data_stored":6891824762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8878268416,"data_stored":11034007864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3332874240,"data_stored":4142846654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:10:59.730 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:59.730 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:59.731 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:59.731 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:59.731 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:59.732 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:59.732 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:11:00.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:00.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:00.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:00.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:00.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:00.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:03.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:03.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:03.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:03.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:03.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:03.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:03.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:03.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:04.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:04.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:04.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:04.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:04.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:04.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:04.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:04.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:04.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:04.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:04.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:04.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:05.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:05.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:05.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:05.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:05.891 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:05.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:08.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:08.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:08.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:08.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:08.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:08.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:08.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:08.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:08.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:08.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:09.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:09.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:09.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:09.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:11.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:11.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:11.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:11.238+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:12.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:12.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:12.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:12.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:12.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:12.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:12.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:12.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:12.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:12.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:12.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:12.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:13.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:13.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:11:13.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:13.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:13.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:13.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:13.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:13.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:13.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:13.846+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:13.854 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:13.846+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:13.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:13.846+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:15.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:15.562+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:15.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:15.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:15.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:15.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:15.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:15.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:15.781 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:15.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:11:15.782 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:15.778+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:15.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:15.790+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:15.804 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:15.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:15.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:15.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:15.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:15.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:15.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:15.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:17.396 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:17.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:17.408 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:17.406+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:17.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:17.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:17.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:17.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:17.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:17.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:18.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:18.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:18.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:18.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:19.196 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:19.186+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:19.212 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:19.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:19.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:19.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:19.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:19.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:19.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:19.242+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:19.733 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:20.494 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:20.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:20.513 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:20.510+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:20.697 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:11:20.697 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:11:20.771 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2395,"stamp":"2023-10-14T01:11:19.628399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28126886744,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2219669,"num_read_kb":8630075,"num_write":2176290,"num_write_kb":8705561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80494,"ondisk_log_size":80494,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22972836,"kb_used_data":17409288,"kb_used_omap":1211,"kb_used_meta":5562308,"kb_avail":260142684,"statfs":{"total":289910292480,"available":266386108416,"internally_reserved":0,"allocated":17827110912,"data_stored":22137236752,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1240285,"internal_metadata":5695804195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,4,7,12,8,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17129472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4123,"num_read_kb":16444,"num_write":4167,"num_write_kb":16668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008516"},"pg_stats":[{"pgid":"2.7","version":"142'291922","reported_seq":573503,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:16.640067+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:11:16.640067+0000","last_peered":"2023-10-14T01:11:16.640067+0000","last_clean":"2023-10-14T01:11:16.640067+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:11:16.640067+0000","last_undegraded":"2023-10-14T01:11:16.640067+0000","last_fullsized":"2023-10-14T01:11:16.640067+0000","mapping_epoch":13,"log_start":"139'281902","ondisk_log_start":"139'281902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3557454262,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":282997,"num_read_kb":1094711,"num_write":274850,"num_write_kb":1099158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'291983","reported_seq":574077,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:16.647559+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:11:16.647559+0000","last_peered":"2023-10-14T01:11:16.647559+0000","last_clean":"2023-10-14T01:11:16.647559+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:11:16.647559+0000","last_undegraded":"2023-10-14T01:11:16.647559+0000","last_fullsized":"2023-10-14T01:11:16.647559+0000","mapping_epoch":13,"log_start":"139'281910","ondisk_log_start":"139'281910","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":73,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 48s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3550495088,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":285092,"num_read_kb":1093109,"num_write":274892,"num_write_kb":1099195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'263936","reported_seq":525846,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:16.623003+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:11:16.623003+0000","last_peered":"2023-10-14T01:11:16.623003+0000","last_clean":"2023-10-14T01:11:16.623003+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:11:16.623003+0000","last_undegraded":"2023-10-14T01:11:16.623003+0000","last_fullsized":"2023-10-14T01:11:16.623003+0000","mapping_epoch":13,"log_start":"138'253905","ondisk_log_start":"138'253905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3396452754,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":262338,"num_read_kb":1033672,"num_write":263318,"num_write_kb":1053242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'257751","reported_seq":516533,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:16.602495+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:11:16.602495+0000","last_peered":"2023-10-14T01:11:16.602495+0000","last_clean":"2023-10-14T01:11:16.602495+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:11:16.602495+0000","last_undegraded":"2023-10-14T01:11:16.602495+0000","last_fullsized":"2023-10-14T01:11:16.602495+0000","mapping_epoch":13,"log_start":"138'247704","ondisk_log_start":"138'247704","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3324223619,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":259249,"num_read_kb":1019521,"num_write":257146,"num_write_kb":1028506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"142'291240","reported_seq":583471,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:17.594568+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:11:17.594568+0000","last_peered":"2023-10-14T01:11:17.594568+0000","last_clean":"2023-10-14T01:11:17.594568+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:11:17.594568+0000","last_undegraded":"2023-10-14T01:11:17.594568+0000","last_fullsized":"2023-10-14T01:11:17.594568+0000","mapping_epoch":13,"log_start":"139'281208","ondisk_log_start":"139'281208","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1177,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 563s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3739373887,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":292738,"num_read_kb":1145418,"num_write":288982,"num_write_kb":1155781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"142'280755","reported_seq":557454,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:15.193561+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:11:15.193561+0000","last_peered":"2023-10-14T01:11:15.193561+0000","last_clean":"2023-10-14T01:11:15.193561+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:11:15.193561+0000","last_undegraded":"2023-10-14T01:11:15.193561+0000","last_fullsized":"2023-10-14T01:11:15.193561+0000","mapping_epoch":13,"log_start":"138'270709","ondisk_log_start":"138'270709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3507663117,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":277850,"num_read_kb":1074980,"num_write":272045,"num_write_kb":1087966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"142'284940","reported_seq":568502,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:15.149264+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:11:15.149264+0000","last_peered":"2023-10-14T01:11:15.149264+0000","last_clean":"2023-10-14T01:11:15.149264+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:11:15.149264+0000","last_undegraded":"2023-10-14T01:11:15.149264+0000","last_fullsized":"2023-10-14T01:11:15.149264+0000","mapping_epoch":13,"log_start":"139'274911","ondisk_log_start":"139'274911","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3556167929,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":284922,"num_read_kb":1097080,"num_write":274423,"num_write_kb":1097418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"142'279496","reported_seq":552998,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:16.598153+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:11:16.598153+0000","last_peered":"2023-10-14T01:11:16.598153+0000","last_clean":"2023-10-14T01:11:16.598153+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:11:16.598153+0000","last_undegraded":"2023-10-14T01:11:16.598153+0000","last_fullsized":"2023-10-14T01:11:16.598153+0000","mapping_epoch":13,"log_start":"139'269407","ondisk_log_start":"139'269407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":198,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 55s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3494465720,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":274357,"num_read_kb":1071475,"num_write":270465,"num_write_kb":1081747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":349,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:44.334764+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:09:44.334764+0000","last_peered":"2023-10-14T01:09:44.334764+0000","last_clean":"2023-10-14T01:09:44.334764+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:09:44.334764+0000","last_undegraded":"2023-10-14T01:09:44.334764+0000","last_fullsized":"2023-10-14T01:09:44.334764+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28126296376,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2219543,"num_read_kb":8629966,"num_write":2176121,"num_write_kb":8703013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17824694272,"data_stored":22136173168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80367,"ondisk_log_size":80367,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5259360,"kb_used_data":3268184,"kb_used_omap":214,"kb_used_meta":1990953,"kb_avail":89112480,"statfs":{"total":96636764160,"available":91251179520,"internally_reserved":0,"allocated":3346620416,"data_stored":4155786462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":2038736484},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.86599999999999999,"15min":0.81399999999999995},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.5460000000000003,"5min":7.492,"15min":9.4209999999999994},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.88800000000000001,"15min":0.89400000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.19900000000000001},"max":{"1min":6.2080000000000002,"5min":7.415,"15min":10.847},"last":6.2080000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:11:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.163,"15min":1.2789999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.191},"max":{"1min":8.4939999999999998,"5min":8.4939999999999998,"15min":14.103999999999999},"last":0.38},{"interface":"front","average":{"1min":1.9370000000000001,"5min":1.214,"15min":1.206},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":9.2899999999999991,"5min":9.2899999999999991,"15min":12.928000000000001},"last":6.2530000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739257,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10656124,"kb_used_data":8703472,"kb_used_omap":623,"kb_used_meta":1952016,"kb_avail":83715716,"statfs":{"total":96636764160,"available":85724893184,"internally_reserved":0,"allocated":8912355328,"data_stored":11067821432,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":1998864873},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,6,2,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:26 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":0.65600000000000003,"15min":0.75700000000000001},"min":{"1min":0.20699999999999999,"5min":0.19400000000000001,"15min":0.16800000000000001},"max":{"1min":10.552,"5min":10.552,"15min":12.369999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.1910000000000001,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.245,"5min":0.24199999999999999,"15min":0.192},"max":{"1min":10.52,"5min":10.52,"15min":12.407},"last":0.31}]},{"osd":2,"last update":"Sat Oct 14 01:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.3540000000000001,"15min":1.169},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":8.9469999999999992,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.374},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.423,"15min":1.1160000000000001},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":5.3170000000000002,"5min":39.412999999999997,"15min":39.412999999999997},"last":4.633}]}]},{"osd":0,"up_from":8,"seq":34359739268,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7057352,"kb_used_data":5437632,"kb_used_omap":373,"kb_used_meta":1619338,"kb_avail":87314488,"statfs":{"total":96636764160,"available":89410035712,"internally_reserved":0,"allocated":5568135168,"data_stored":6913628858,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1658202838},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.65000000000000002,"15min":0.93700000000000006},"min":{"1min":0.245,"5min":0.182,"15min":0.17699999999999999},"max":{"1min":0.97399999999999998,"5min":6.7290000000000001,"15min":12.855},"last":0.376},{"interface":"front","average":{"1min":0.315,"5min":0.56200000000000006,"15min":0.84199999999999997},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":0.65700000000000003,"5min":9.2439999999999998,"15min":16.655000000000001},"last":5.7670000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:10:21 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.208,"15min":1.4059999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":8.0060000000000002,"5min":10.539999999999999,"15min":10.539999999999999},"last":5.9950000000000001},{"interface":"front","average":{"1min":1.9510000000000001,"5min":0.94699999999999995,"15min":0.80700000000000005},"min":{"1min":0.253,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":8.9710000000000001,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":3.5550000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5567102976,"data_stored":6912870010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8912293888,"data_stored":11068033336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3345297408,"data_stored":4155269822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:11:20.773 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:20.773 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:20.774 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:20.774 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:20.774 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:20.775 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:20.775 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:11:20.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:20.994+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:21.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:21.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:21.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:21.026+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:22.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:22.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:22.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:22.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:22.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:22.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:22.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:22.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:22.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:22.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:22.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:23.216 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:23.210+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:23.216 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:23.210+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:23.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:23.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:23.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:23.718+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:24.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:11:24.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:24.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:24.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:24.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:24.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:24.385 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:24.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:24.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:24.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:24.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:24.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:25.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:25.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:25.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:25.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:25.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:25.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:25.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:25.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:25.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:25.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:25.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:25.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:25.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:25.887 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:25.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:25.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:25.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:25.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:25.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:25.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:25.918+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:26.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:26.970+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:26.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:26.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:26.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:26.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:26.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:26.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:26.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:26.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:26.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:26.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:27.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:27.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:27.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:27.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:27.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:27.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:11:27.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:27.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:27.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:27.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:27.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:27.754+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:27.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:27.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:27.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:27.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:27.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:27.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:28.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:28.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:28.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:28.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:28.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:28.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:28.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:28.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:28.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:29.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:29.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:29.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:29.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:32.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:32.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:32.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:32.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:32.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:32.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:32.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:32.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:33.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:33.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:11:33.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:33.774+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:33.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:33.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:33.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:33.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:33.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:33.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:34.229 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.226+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:11:34.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:11:34.271 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:34.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:34.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:34.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:34.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:34.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:34.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:11:34.712 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:34.754 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:34.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:34.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:34.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:34.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:34.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:34.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:34.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:34.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:34.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:34.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:34.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:34.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:34.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:35.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:35.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:35.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:35.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:35.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:35.114+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:38.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:38.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:38.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:38.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:38.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:38.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:38.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:38.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:38.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:38.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:38.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:38.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:38.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:38.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:40.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:40.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:40.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:40.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:40.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:40.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:40.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:40.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:40.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:40.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:40.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:40.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:40.776 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:41.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:41.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:11:41.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:41.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:11:41.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:41.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:41.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:41.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:41.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:41.285+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:41.542 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:11:41.543 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:11:41.595 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2405,"stamp":"2023-10-14T01:11:39.631990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28160318296,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2227874,"num_read_kb":8662795,"num_write":2184378,"num_write_kb":8737913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80444,"ondisk_log_size":80444,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22755140,"kb_used_data":17474856,"kb_used_omap":1211,"kb_used_meta":5279044,"kb_avail":260360380,"statfs":{"total":289910292480,"available":266609029120,"internally_reserved":0,"allocated":17894252544,"data_stored":22204378384,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1240285,"internal_metadata":5405741859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,7,7,9,15,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18190336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4524,"num_read_kb":18048,"num_write":4386,"num_write_kb":17544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002072"},"pg_stats":[{"pgid":"2.7","version":"142'293017","reported_seq":575669,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.112192+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:11:37.112192+0000","last_peered":"2023-10-14T01:11:37.112192+0000","last_clean":"2023-10-14T01:11:37.112192+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:11:37.112192+0000","last_undegraded":"2023-10-14T01:11:37.112192+0000","last_fullsized":"2023-10-14T01:11:37.112192+0000","mapping_epoch":13,"log_start":"139'283002","ondisk_log_start":"139'283002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3561718198,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":284052,"num_read_kb":1098927,"num_write":275876,"num_write_kb":1103262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"142'293028","reported_seq":576233,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:37.122179+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:11:37.122179+0000","last_peered":"2023-10-14T01:11:37.122179+0000","last_clean":"2023-10-14T01:11:37.122179+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:11:37.122179+0000","last_undegraded":"2023-10-14T01:11:37.122179+0000","last_fullsized":"2023-10-14T01:11:37.122179+0000","mapping_epoch":13,"log_start":"139'283010","ondisk_log_start":"139'283010","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":104,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 69s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3554488688,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":286180,"num_read_kb":1097441,"num_write":275856,"num_write_kb":1103051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"142'264904","reported_seq":527766,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.110499+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:11:37.110499+0000","last_peered":"2023-10-14T01:11:37.110499+0000","last_clean":"2023-10-14T01:11:37.110499+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:11:37.110499+0000","last_undegraded":"2023-10-14T01:11:37.110499+0000","last_fullsized":"2023-10-14T01:11:37.110499+0000","mapping_epoch":13,"log_start":"139'254805","ondisk_log_start":"139'254805","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3400434066,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":263291,"num_read_kb":1037472,"num_write":264286,"num_write_kb":1057114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'258715","reported_seq":518431,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.073437+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:11:37.073437+0000","last_peered":"2023-10-14T01:11:37.073437+0000","last_clean":"2023-10-14T01:11:37.073437+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:11:37.073437+0000","last_undegraded":"2023-10-14T01:11:37.073437+0000","last_fullsized":"2023-10-14T01:11:37.073437+0000","mapping_epoch":13,"log_start":"138'248704","ondisk_log_start":"138'248704","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3328249987,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":260183,"num_read_kb":1023249,"num_write":258110,"num_write_kb":1032362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"142'292330","reported_seq":585626,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:37.904409+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:11:37.904409+0000","last_peered":"2023-10-14T01:11:37.904409+0000","last_clean":"2023-10-14T01:11:37.904409+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:11:37.904409+0000","last_undegraded":"2023-10-14T01:11:37.904409+0000","last_fullsized":"2023-10-14T01:11:37.904409+0000","mapping_epoch":13,"log_start":"139'282308","ondisk_log_start":"139'282308","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1225,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 583s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3743764799,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":293789,"num_read_kb":1149602,"num_write":290041,"num_write_kb":1160017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"142'281840","reported_seq":559575,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:35.352563+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:11:35.352563+0000","last_peered":"2023-10-14T01:11:35.352563+0000","last_clean":"2023-10-14T01:11:35.352563+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:11:35.352563+0000","last_undegraded":"2023-10-14T01:11:35.352563+0000","last_fullsized":"2023-10-14T01:11:35.352563+0000","mapping_epoch":13,"log_start":"139'271809","ondisk_log_start":"139'271809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3511894285,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":278868,"num_read_kb":1079044,"num_write":273074,"num_write_kb":1092082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"142'286073","reported_seq":570705,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:35.337482+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:11:35.337482+0000","last_peered":"2023-10-14T01:11:35.337482+0000","last_clean":"2023-10-14T01:11:35.337482+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:11:35.337482+0000","last_undegraded":"2023-10-14T01:11:35.337482+0000","last_fullsized":"2023-10-14T01:11:35.337482+0000","mapping_epoch":13,"log_start":"139'276011","ondisk_log_start":"139'276011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3560534265,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":285974,"num_read_kb":1101280,"num_write":275486,"num_write_kb":1101670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"142'280566","reported_seq":555137,"reported_epoch":142,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:37.093971+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:11:37.093971+0000","last_peered":"2023-10-14T01:11:37.093971+0000","last_clean":"2023-10-14T01:11:37.093971+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:11:37.093971+0000","last_undegraded":"2023-10-14T01:11:37.093971+0000","last_fullsized":"2023-10-14T01:11:37.093971+0000","mapping_epoch":13,"log_start":"139'270507","ondisk_log_start":"139'270507","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":238,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 75s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3498643640,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":275411,"num_read_kb":1075671,"num_write":271480,"num_write_kb":1085807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":349,"reported_epoch":141,"state":"active+clean","last_fresh":"2023-10-14T01:09:44.334764+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:09:44.334764+0000","last_peered":"2023-10-14T01:09:44.334764+0000","last_clean":"2023-10-14T01:09:44.334764+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:09:44.334764+0000","last_undegraded":"2023-10-14T01:09:44.334764+0000","last_fullsized":"2023-10-14T01:09:44.334764+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28159727928,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2227748,"num_read_kb":8662686,"num_write":2184209,"num_write_kb":8735365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17890922496,"data_stored":22202401392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80317,"ondisk_log_size":80317,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4802200,"kb_used_data":3280464,"kb_used_omap":214,"kb_used_meta":1521513,"kb_avail":89569640,"statfs":{"total":96636764160,"available":91719311360,"internally_reserved":0,"allocated":3359195136,"data_stored":4168361182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1558029924},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,2,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.86599999999999999,"15min":0.81399999999999995},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.5460000000000003,"5min":7.492,"15min":9.4209999999999994},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.88800000000000001,"15min":0.89400000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.19900000000000001},"max":{"1min":6.2080000000000002,"5min":7.415,"15min":10.847},"last":0.32800000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:11:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.163,"15min":1.2789999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.191},"max":{"1min":8.4939999999999998,"5min":8.4939999999999998,"15min":14.103999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.9370000000000001,"5min":1.214,"15min":1.206},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":9.2899999999999991,"5min":9.2899999999999991,"15min":12.928000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739261,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10817756,"kb_used_data":8737236,"kb_used_omap":623,"kb_used_meta":2079888,"kb_avail":83554084,"statfs":{"total":96636764160,"available":85559382016,"internally_reserved":0,"allocated":8946929664,"data_stored":11102395768,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":2129805801},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,5,3,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.67800000000000005,"15min":0.76400000000000001},"min":{"1min":0.29099999999999998,"5min":0.20200000000000001,"15min":0.16800000000000001},"max":{"1min":1.0900000000000001,"5min":10.552,"15min":12.369999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.72299999999999998,"15min":0.78600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.192},"max":{"1min":0.93500000000000005,"5min":10.52,"15min":12.407},"last":0.67100000000000004}]},{"osd":2,"last update":"Sat Oct 14 01:10:43 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.3540000000000001,"15min":1.169},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":8.9469999999999992,"5min":43.871000000000002,"15min":43.871000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.423,"15min":1.1160000000000001},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":5.3170000000000002,"5min":39.412999999999997,"15min":39.412999999999997},"last":0.34200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739272,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7135184,"kb_used_data":5457156,"kb_used_omap":373,"kb_used_meta":1677642,"kb_avail":87236656,"statfs":{"total":96636764160,"available":89330335744,"internally_reserved":0,"allocated":5588127744,"data_stored":6933621434,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1717906134},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.67500000000000004,"15min":0.93300000000000005},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.5380000000000003,"5min":9.5380000000000003,"15min":12.855},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60199999999999998,"15min":0.76100000000000001},"min":{"1min":0.23400000000000001,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":5.7670000000000003,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.26100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.147,"15min":1.363},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":5.9950000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.246},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.94999999999999996,"15min":0.81599999999999995},"min":{"1min":0.24299999999999999,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":4.6630000000000003,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5587439616,"data_stored":6933206650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8945610752,"data_stored":11101350200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3357872128,"data_stored":4167844542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:11:41.596 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:41.597 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:41.597 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:41.597 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:41.598 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:41.598 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:41.598 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:11:44.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:11:44.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:44.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:44.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.309+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:44.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:44.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:44.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:44.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:44.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:44.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:44.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:44.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:46.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:46.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:46.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:46.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:46.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:46.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:46.517 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:46.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:47.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:11:47.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:11:47.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:11:47.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:47.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:47.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:47.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:47.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:47.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:47.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:47.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:47.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:47.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:49.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:49.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:11:49.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:49.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:11:49.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:49.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:11:49.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:49.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:49.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:49.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:49.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:49.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:49.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:49.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:50.088 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:50.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:50.124 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:50.121+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:50.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:50.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:50.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:50.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:50.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:50.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:50.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:50.125+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:51.835 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:51.829+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:51.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:51.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:51.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:51.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:51.899 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:51.893+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:53.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:53.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:11:53.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:53.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:11:53.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:53.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:11:53.589 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:53.585+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:11:53.604 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:53.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:11:53.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:53.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:11:53.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:11:53.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:11:57.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:57.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:57.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:57.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:57.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:57.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:11:58.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:58.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:11:58.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:58.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:11:58.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:11:58.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:00.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:00.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:00.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:00.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:00.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:00.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:01.599 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:02.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:02.417+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:02.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:02.461+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:02.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:02.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:02.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:02.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:02.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:02.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:02.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:02.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:02.521 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:12:02.521 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:12:02.566 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2417,"stamp":"2023-10-14T01:12:01.645633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28196744024,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2236586,"num_read_kb":8697531,"num_write":2193224,"num_write_kb":8773297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80499,"ondisk_log_size":80499,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23072388,"kb_used_data":17543464,"kb_used_omap":1211,"kb_used_meta":5527684,"kb_avail":260043132,"statfs":{"total":289910292480,"available":266284167168,"internally_reserved":0,"allocated":17964507136,"data_stored":22274609507,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1240285,"internal_metadata":5660349219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,2,24,17,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17215488,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3995,"num_read_kb":15920,"num_write":4173,"num_write_kb":16692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007415"},"pg_stats":[{"pgid":"2.7","version":"143'294101","reported_seq":577790,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.501554+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:11:57.501554+0000","last_peered":"2023-10-14T01:11:57.501554+0000","last_clean":"2023-10-14T01:11:57.501554+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:11:57.501554+0000","last_undegraded":"2023-10-14T01:11:57.501554+0000","last_fullsized":"2023-10-14T01:11:57.501554+0000","mapping_epoch":13,"log_start":"139'284002","ondisk_log_start":"139'284002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3566088630,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":285073,"num_read_kb":1103003,"num_write":276928,"num_write_kb":1107470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'294112","reported_seq":578361,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:57.532595+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:11:57.532595+0000","last_peered":"2023-10-14T01:11:57.532595+0000","last_clean":"2023-10-14T01:11:57.532595+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:11:57.532595+0000","last_undegraded":"2023-10-14T01:11:57.532595+0000","last_fullsized":"2023-10-14T01:11:57.532595+0000","mapping_epoch":13,"log_start":"139'284110","ondisk_log_start":"139'284110","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":135,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 89s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3558719856,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":287199,"num_read_kb":1101497,"num_write":276889,"num_write_kb":1107183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'265925","reported_seq":529815,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.526608+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:11:57.526608+0000","last_peered":"2023-10-14T01:11:57.526608+0000","last_clean":"2023-10-14T01:11:57.526608+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:11:57.526608+0000","last_undegraded":"2023-10-14T01:11:57.526608+0000","last_fullsized":"2023-10-14T01:11:57.526608+0000","mapping_epoch":13,"log_start":"139'255905","ondisk_log_start":"139'255905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3404698002,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":264319,"num_read_kb":1041572,"num_write":265307,"num_write_kb":1061198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'259709","reported_seq":520335,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.457714+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:11:57.457714+0000","last_peered":"2023-10-14T01:11:57.457714+0000","last_clean":"2023-10-14T01:11:57.457714+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:11:57.457714+0000","last_undegraded":"2023-10-14T01:11:57.457714+0000","last_fullsized":"2023-10-14T01:11:57.457714+0000","mapping_epoch":13,"log_start":"139'249704","ondisk_log_start":"139'249704","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3332350083,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":261092,"num_read_kb":1026877,"num_write":259104,"num_write_kb":1036338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"143'293459","reported_seq":587850,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:58.058459+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:11:58.058459+0000","last_peered":"2023-10-14T01:11:58.058459+0000","last_clean":"2023-10-14T01:11:58.058459+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:11:58.058459+0000","last_undegraded":"2023-10-14T01:11:58.058459+0000","last_fullsized":"2023-10-14T01:11:58.058459+0000","mapping_epoch":13,"log_start":"139'283409","ondisk_log_start":"139'283409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1287,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 603s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3748262207,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":294873,"num_read_kb":1153914,"num_write":291132,"num_write_kb":1164381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"143'283151","reported_seq":562224,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:00.569311+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:12:00.569311+0000","last_peered":"2023-10-14T01:12:00.569311+0000","last_clean":"2023-10-14T01:12:00.569311+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:12:00.569311+0000","last_undegraded":"2023-10-14T01:12:00.569311+0000","last_fullsized":"2023-10-14T01:12:00.569311+0000","mapping_epoch":13,"log_start":"139'273109","ondisk_log_start":"139'273109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3517227277,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":280188,"num_read_kb":1084304,"num_write":274365,"num_write_kb":1097246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"143'287407","reported_seq":573344,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:00.565389+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:12:00.565389+0000","last_peered":"2023-10-14T01:12:00.565389+0000","last_clean":"2023-10-14T01:12:00.565389+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:12:00.565389+0000","last_undegraded":"2023-10-14T01:12:00.565389+0000","last_fullsized":"2023-10-14T01:12:00.565389+0000","mapping_epoch":13,"log_start":"139'277311","ondisk_log_start":"139'277311","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3565834489,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":287250,"num_read_kb":1106364,"num_write":276795,"num_write_kb":1106906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"143'281665","reported_seq":557310,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:57.502373+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:11:57.502373+0000","last_peered":"2023-10-14T01:11:57.502373+0000","last_clean":"2023-10-14T01:11:57.502373+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:11:57.502373+0000","last_undegraded":"2023-10-14T01:11:57.502373+0000","last_fullsized":"2023-10-14T01:11:57.502373+0000","mapping_epoch":13,"log_start":"139'271607","ondisk_log_start":"139'271607","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":281,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 96s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3502973112,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":276466,"num_read_kb":1079891,"num_write":272535,"num_write_kb":1090027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":351,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:47.277469+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:11:47.277469+0000","last_peered":"2023-10-14T01:11:47.277469+0000","last_clean":"2023-10-14T01:11:47.277469+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:11:47.277469+0000","last_undegraded":"2023-10-14T01:11:47.277469+0000","last_fullsized":"2023-10-14T01:11:47.277469+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28196153656,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2236460,"num_read_kb":8697422,"num_write":2193055,"num_write_kb":8770749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17962110976,"data_stored":22273589872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80372,"ondisk_log_size":80372,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739278,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4867284,"kb_used_data":3295564,"kb_used_omap":214,"kb_used_meta":1571497,"kb_avail":89504556,"statfs":{"total":96636764160,"available":91652665344,"internally_reserved":0,"allocated":3374657536,"data_stored":4183815759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1609213540},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:13 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.86599999999999999,"15min":0.81399999999999995},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":4.5460000000000003,"5min":7.492,"15min":9.4209999999999994},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.88800000000000001,"15min":0.89400000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.19900000000000001},"max":{"1min":6.2080000000000002,"5min":7.415,"15min":10.847},"last":0.23799999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:11:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.163,"15min":1.2789999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.191},"max":{"1min":8.4939999999999998,"5min":8.4939999999999998,"15min":14.103999999999999},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.9370000000000001,"5min":1.214,"15min":1.206},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":9.2899999999999991,"5min":9.2899999999999991,"15min":12.928000000000001},"last":0.32800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739265,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10946592,"kb_used_data":8770324,"kb_used_omap":623,"kb_used_meta":2175632,"kb_avail":83425248,"statfs":{"total":96636764160,"available":85427453952,"internally_reserved":0,"allocated":8980811776,"data_stored":11136270057,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":638487,"internal_metadata":2227847657},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,8,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.67800000000000005,"15min":0.76400000000000001},"min":{"1min":0.29099999999999998,"5min":0.20200000000000001,"15min":0.16800000000000001},"max":{"1min":1.0900000000000001,"5min":10.552,"15min":12.369999999999999},"last":0.433},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.72299999999999998,"15min":0.78600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.192},"max":{"1min":0.93500000000000005,"5min":10.52,"15min":12.407},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:11:47 2023","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.1759999999999999,"15min":1.226},"min":{"1min":0.27400000000000002,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":8.7780000000000005,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.3149999999999999,"15min":1.24},"min":{"1min":0.255,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":8.6489999999999991,"5min":8.6489999999999991,"15min":39.412999999999997},"last":11.412000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739276,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7258512,"kb_used_data":5477576,"kb_used_omap":373,"kb_used_meta":1780554,"kb_avail":87113328,"statfs":{"total":96636764160,"available":89204047872,"internally_reserved":0,"allocated":5609037824,"data_stored":6954523691,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1823288022},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,8,2,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.67500000000000004,"15min":0.93300000000000005},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.5380000000000003,"5min":9.5380000000000003,"15min":12.855},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60199999999999998,"15min":0.76100000000000001},"min":{"1min":0.23400000000000001,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":5.7670000000000003,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.23200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.147,"15min":1.363},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":5.9950000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.94999999999999996,"15min":0.81599999999999995},"min":{"1min":0.24299999999999999,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":4.6630000000000003,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5608402944,"data_stored":6954169978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8979828736,"data_stored":11135568184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3373879296,"data_stored":4183851710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:12:02.567 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:02.568 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:02.568 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:02.568 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:02.569 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:02.569 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:02.570 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:03.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:03.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:03.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:03.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:03.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:03.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:03.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:03.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:03.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:03.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:03.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:03.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:03.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:03.125+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:05.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:05.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:05.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:05.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:05.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:05.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:05.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:05.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:05.723 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:05.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:12:05.723 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:05.721+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:05.732 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:05.733+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:05.741 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:05.741+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:05.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:05.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:05.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:05.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:05.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:05.749+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:05.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:05.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:05.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:05.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:06.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:06.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:06.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:06.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:06.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:06.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:06.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:06.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:06.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:06.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:06.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:06.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:06.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:06.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:09.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:09.801+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:12:09.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:09.801+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:09.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:09.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:09.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:09.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:09.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:09.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:09.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:09.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:09.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:09.845+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:10.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:10.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:10.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:10.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:10.323 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:10.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:12:10.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.321+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:12:10.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:12:10.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:10.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:10.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:10.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:10.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:10.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:10.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:10.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:10.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:10.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:10.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:10.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:10.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:11.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:11.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:12:11.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:11.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:11.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:11.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:13.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:13.097+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:13.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:13.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:13.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:13.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:13.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:13.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:13.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:13.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:12:13.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:13.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:13.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:13.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:14.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:14.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:15.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:15.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:15.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:15.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:15.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:15.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:16.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:16.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:16.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:12:16.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:16.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:16.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:16.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:16.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:17.000 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:16.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:17.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:17.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:17.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:17.473+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:17.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:17.481+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:17.495 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:17.493+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:17.514 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:17.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:17.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:17.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:17.515 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:17.513+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:18.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:12:18.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:12:18.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:18.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:18.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:18.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:18.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:18.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:18.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:18.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:18.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:18.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:18.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:22.571 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:22.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:22.877+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:22.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:22.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:22.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:22.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:22.949 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:22.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:22.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:22.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:23.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:22.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:23.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:23.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:23.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:23.201+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:12:23.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:23.201+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:23.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:23.209+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:23.217 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:23.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:23.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:23.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:23.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:23.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:23.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:23.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:23.398 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:12:23.399 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:12:23.437 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2427,"stamp":"2023-10-14T01:12:21.649021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28230675288,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2244878,"num_read_kb":8730615,"num_write":2201447,"num_write_kb":8806189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80605,"ondisk_log_size":80605,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22739736,"kb_used_data":17608512,"kb_used_omap":1277,"kb_used_meta":5129922,"kb_avail":260375784,"statfs":{"total":289910292480,"available":266624802816,"internally_reserved":0,"allocated":18031116288,"data_stored":22341218659,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1308039,"internal_metadata":5253040761},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,5,10,12,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19181568,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4530,"num_read_kb":18068,"num_write":4643,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002074"},"pg_stats":[{"pgid":"2.7","version":"143'295201","reported_seq":579995,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:18.116352+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:12:18.116352+0000","last_peered":"2023-10-14T01:12:18.116352+0000","last_clean":"2023-10-14T01:12:18.116352+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:12:18.116352+0000","last_undegraded":"2023-10-14T01:12:18.116352+0000","last_fullsized":"2023-10-14T01:12:18.116352+0000","mapping_epoch":13,"log_start":"139'285102","ondisk_log_start":"139'285102","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3570356662,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":286161,"num_read_kb":1107343,"num_write":277956,"num_write_kb":1111582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'295210","reported_seq":580556,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:18.123898+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:12:18.123898+0000","last_peered":"2023-10-14T01:12:18.123898+0000","last_clean":"2023-10-14T01:12:18.123898+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:12:18.123898+0000","last_undegraded":"2023-10-14T01:12:18.123898+0000","last_fullsized":"2023-10-14T01:12:18.123898+0000","mapping_epoch":13,"log_start":"139'285110","ondisk_log_start":"139'285110","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":185,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 110s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3562840432,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":288269,"num_read_kb":1105765,"num_write":277887,"num_write_kb":1111175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'266939","reported_seq":531797,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:18.115704+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:12:18.115704+0000","last_peered":"2023-10-14T01:12:18.115704+0000","last_clean":"2023-10-14T01:12:18.115704+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:12:18.115704+0000","last_undegraded":"2023-10-14T01:12:18.115704+0000","last_fullsized":"2023-10-14T01:12:18.115704+0000","mapping_epoch":13,"log_start":"139'256905","ondisk_log_start":"139'256905","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3408937362,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":265285,"num_read_kb":1045424,"num_write":266321,"num_write_kb":1065254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'260671","reported_seq":522262,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:18.115589+0000","last_change":"2023-10-13T23:57:08.678158+0000","last_active":"2023-10-14T01:12:18.115589+0000","last_peered":"2023-10-14T01:12:18.115589+0000","last_clean":"2023-10-14T01:12:18.115589+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:12:18.115589+0000","last_undegraded":"2023-10-14T01:12:18.115589+0000","last_fullsized":"2023-10-14T01:12:18.115589+0000","mapping_epoch":13,"log_start":"139'250604","ondisk_log_start":"139'250604","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3336310915,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":262058,"num_read_kb":1030737,"num_write":260066,"num_write_kb":1040186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"143'294542","reported_seq":590066,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:18.204971+0000","last_change":"2023-10-14T01:01:54.591863+0000","last_active":"2023-10-14T01:12:18.204971+0000","last_peered":"2023-10-14T01:12:18.204971+0000","last_clean":"2023-10-14T01:12:18.204971+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:12:18.204971+0000","last_undegraded":"2023-10-14T01:12:18.204971+0000","last_fullsized":"2023-10-14T01:12:18.204971+0000","mapping_epoch":13,"log_start":"139'284509","ondisk_log_start":"139'284509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1321,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 623s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3752616255,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":296000,"num_read_kb":1158414,"num_write":292195,"num_write_kb":1168633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"143'284220","reported_seq":564317,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:20.868088+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:12:20.868088+0000","last_peered":"2023-10-14T01:12:20.868088+0000","last_clean":"2023-10-14T01:12:20.868088+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:12:20.868088+0000","last_undegraded":"2023-10-14T01:12:20.868088+0000","last_fullsized":"2023-10-14T01:12:20.868088+0000","mapping_epoch":13,"log_start":"139'274209","ondisk_log_start":"139'274209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3521556749,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":281193,"num_read_kb":1088312,"num_write":275418,"num_write_kb":1101458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"143'288466","reported_seq":575469,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:20.869833+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:12:20.869833+0000","last_peered":"2023-10-14T01:12:20.869833+0000","last_clean":"2023-10-14T01:12:20.869833+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:12:20.869833+0000","last_undegraded":"2023-10-14T01:12:20.869833+0000","last_fullsized":"2023-10-14T01:12:20.869833+0000","mapping_epoch":13,"log_start":"139'278411","ondisk_log_start":"139'278411","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3570086137,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":288299,"num_read_kb":1110548,"num_write":277834,"num_write_kb":1111062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"143'282786","reported_seq":559468,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:18.090520+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:12:18.090520+0000","last_peered":"2023-10-14T01:12:18.090520+0000","last_clean":"2023-10-14T01:12:18.090520+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:12:18.090520+0000","last_undegraded":"2023-10-14T01:12:18.090520+0000","last_fullsized":"2023-10-14T01:12:18.090520+0000","mapping_epoch":13,"log_start":"139'272707","ondisk_log_start":"139'272707","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":320,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 116s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3507380408,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":277487,"num_read_kb":1083963,"num_write":273601,"num_write_kb":1094291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":351,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:47.277469+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:11:47.277469+0000","last_peered":"2023-10-14T01:11:47.277469+0000","last_clean":"2023-10-14T01:11:47.277469+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:11:47.277469+0000","last_undegraded":"2023-10-14T01:11:47.277469+0000","last_fullsized":"2023-10-14T01:11:47.277469+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28230084920,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2244752,"num_read_kb":8730506,"num_write":2201278,"num_write_kb":8803641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4572,"num_omap_keys":90,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18029187072,"data_stored":22340665968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80478,"ondisk_log_size":80478,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739282,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4916432,"kb_used_data":3308616,"kb_used_omap":214,"kb_used_meta":1607593,"kb_avail":89455408,"statfs":{"total":96636764160,"available":91602337792,"internally_reserved":0,"allocated":3388022784,"data_stored":4197181007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1646175844},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.63800000000000001,"15min":0.747},"min":{"1min":0.20100000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.1150000000000002,"5min":5.5860000000000003,"15min":7.492},"last":3.0089999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91900000000000004,"15min":0.90900000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":4.4619999999999997,"5min":6.6539999999999999,"15min":10.847},"last":0.26500000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:12:14 2023","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.27,"15min":1.3109999999999999},"min":{"1min":0.217,"5min":0.193,"15min":0.191},"max":{"1min":11.077999999999999,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":2.044,"5min":1.5289999999999999,"15min":1.1870000000000001},"min":{"1min":0.249,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":14.586,"5min":14.586,"15min":14.586},"last":0.30099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739269,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10489292,"kb_used_data":8802372,"kb_used_omap":689,"kb_used_meta":1686222,"kb_avail":83882548,"statfs":{"total":96636764160,"available":85895729152,"internally_reserved":0,"allocated":9013628928,"data_stored":11169087209,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":706241,"internal_metadata":1726691647},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,2,8,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.67800000000000005,"15min":0.76400000000000001},"min":{"1min":0.29099999999999998,"5min":0.20200000000000001,"15min":0.16800000000000001},"max":{"1min":1.0900000000000001,"5min":10.552,"15min":12.369999999999999},"last":0.435},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.72299999999999998,"15min":0.78600000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.192},"max":{"1min":0.93500000000000005,"5min":10.52,"15min":12.407},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:11:47 2023","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.1759999999999999,"15min":1.226},"min":{"1min":0.27400000000000002,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":8.7780000000000005,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.3149999999999999,"15min":1.24},"min":{"1min":0.255,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":8.6489999999999991,"5min":8.6489999999999991,"15min":39.412999999999997},"last":0.46000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739280,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7334012,"kb_used_data":5497524,"kb_used_omap":373,"kb_used_meta":1836106,"kb_avail":87037828,"statfs":{"total":96636764160,"available":89126735872,"internally_reserved":0,"allocated":5629464576,"data_stored":6974950443,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1880173270},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.67500000000000004,"15min":0.93300000000000005},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":9.5380000000000003,"5min":9.5380000000000003,"15min":12.855},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.60199999999999998,"15min":0.76100000000000001},"min":{"1min":0.23400000000000001,"5min":0.20899999999999999,"15min":0.17399999999999999},"max":{"1min":5.7670000000000003,"5min":9.2439999999999998,"15min":16.655000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:11:24 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.147,"15min":1.363},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":5.9950000000000001,"5min":10.539999999999999,"15min":10.539999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.94999999999999996,"15min":0.81599999999999995},"min":{"1min":0.24299999999999999,"5min":0.23000000000000001,"15min":0.20399999999999999},"max":{"1min":4.6630000000000003,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.27700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5628784640,"data_stored":6974551674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9013694464,"data_stored":11169433912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3386707968,"data_stored":4196680382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:12:23.439 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:23.439 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:23.440 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:23.440 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:23.441 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:23.441 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:23.441 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:24.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:24.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:24.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:24.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:24.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:24.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:24.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:24.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:24.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:24.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:24.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:24.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:24.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:24.941+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:25.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:25.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:25.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:25.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:25.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:25.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:25.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:25.593+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:25.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:25.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:25.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:25.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:26.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:26.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:26.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:26.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:26.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:26.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:12:26.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:26.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:26.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:26.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:26.797 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:26.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:26.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:26.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:26.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:26.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:26.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:26.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:26.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:26.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:27.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:27.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:27.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:27.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:27.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:27.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:28.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:28.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:28.955 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:28.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:31.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:31.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:31.225 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:31.225+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:31.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:31.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:31.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:31.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:31.257 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:31.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:31.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:31.253+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:32.302 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:32.301+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:32.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:32.337+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:32.353+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:32.353+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:32.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:32.353+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:33.286 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:33.299 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:33.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.296+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:33.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:33.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:33.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:33.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:33.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:33.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:33.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:35.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:35.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:12:35.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:35.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:35.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:35.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:35.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:35.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:37.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:37.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:12:37.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:37.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:37.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:37.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:37.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:37.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:37.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:37.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:37.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:37.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:37.246 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:37.244+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:39.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:39.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:39.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:39.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:39.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:39.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:39.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:39.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:39.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:39.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:39.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:39.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:40.789 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:40.788+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:40.806 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:40.804+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:40.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:40.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:12:40.812 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:40.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:40.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:40.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:40.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:40.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:40.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:40.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:40.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:40.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:40.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:40.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:40.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:40.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:40.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:40.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:40.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:40.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:41.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:41.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:41.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:41.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:41.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:41.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:41.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:41.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:41.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:41.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:41.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:41.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:42.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:42.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:12:42.979 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:42.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:42.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:12:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:42.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:12:42.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:42.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:12:43.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:43.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:43.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:43.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:43.085 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:43.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:43.442 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:43.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:43.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:43.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:43.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:43.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:43.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:43.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:43.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:44.318 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:12:44.319 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:12:44.371 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2438,"stamp":"2023-10-14T01:12:43.655460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28267477848,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2253706,"num_read_kb":8765827,"num_write":2210393,"num_write_kb":8841973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80502,"ondisk_log_size":80502,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23012352,"kb_used_data":17679648,"kb_used_omap":1277,"kb_used_meta":5331394,"kb_avail":260103168,"statfs":{"total":289910292480,"available":266345644032,"internally_reserved":0,"allocated":18103959552,"data_stored":22414061923,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1308039,"internal_metadata":5459348089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,5,12,11,14,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20238336,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4901,"num_read_kb":19556,"num_write":4910,"num_write_kb":19640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004743"},"pg_stats":[{"pgid":"2.7","version":"143'296572","reported_seq":582563,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:43.632816+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:12:43.632816+0000","last_peered":"2023-10-14T01:12:43.632816+0000","last_clean":"2023-10-14T01:12:43.632816+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:12:43.632816+0000","last_undegraded":"2023-10-14T01:12:43.632816+0000","last_fullsized":"2023-10-14T01:12:43.632816+0000","mapping_epoch":13,"log_start":"141'286502","ondisk_log_start":"141'286502","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3575603638,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":287339,"num_read_kb":1112047,"num_write":279226,"num_write_kb":1116662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'296629","reported_seq":583252,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:43.621686+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:12:43.621686+0000","last_peered":"2023-10-14T01:12:43.621686+0000","last_clean":"2023-10-14T01:12:43.621686+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:12:43.621686+0000","last_undegraded":"2023-10-14T01:12:43.621686+0000","last_fullsized":"2023-10-14T01:12:43.621686+0000","mapping_epoch":13,"log_start":"141'286610","ondisk_log_start":"141'286610","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":250,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 135s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3568161136,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":289516,"num_read_kb":1110733,"num_write":279177,"num_write_kb":1116335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'268166","reported_seq":534209,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:43.603987+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:12:43.603987+0000","last_peered":"2023-10-14T01:12:43.603987+0000","last_clean":"2023-10-14T01:12:43.603987+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:12:43.603987+0000","last_undegraded":"2023-10-14T01:12:43.603987+0000","last_fullsized":"2023-10-14T01:12:43.603987+0000","mapping_epoch":13,"log_start":"140'258105","ondisk_log_start":"140'258105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3414004114,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":266473,"num_read_kb":1050156,"num_write":267548,"num_write_kb":1070162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'261825","reported_seq":524505,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:43.579514+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:12:43.579514+0000","last_peered":"2023-10-14T01:12:43.579514+0000","last_clean":"2023-10-14T01:12:43.579514+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:12:43.579514+0000","last_undegraded":"2023-10-14T01:12:43.579514+0000","last_fullsized":"2023-10-14T01:12:43.579514+0000","mapping_epoch":13,"log_start":"139'251804","ondisk_log_start":"139'251804","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":28,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3340939395,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":263149,"num_read_kb":1035085,"num_write":261205,"num_write_kb":1044742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"143'295582","reported_seq":592152,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:38.617830+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:12:38.617830+0000","last_peered":"2023-10-14T01:12:38.617830+0000","last_clean":"2023-10-14T01:12:38.617830+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:12:38.617830+0000","last_undegraded":"2023-10-14T01:12:38.617830+0000","last_fullsized":"2023-10-14T01:12:38.617830+0000","mapping_epoch":13,"log_start":"140'285509","ondisk_log_start":"140'285509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3756802367,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":297030,"num_read_kb":1162526,"num_write":293203,"num_write_kb":1172665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"143'285185","reported_seq":566232,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:40.975762+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:12:40.975762+0000","last_peered":"2023-10-14T01:12:40.975762+0000","last_clean":"2023-10-14T01:12:40.975762+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:12:40.975762+0000","last_undegraded":"2023-10-14T01:12:40.975762+0000","last_fullsized":"2023-10-14T01:12:40.975762+0000","mapping_epoch":13,"log_start":"140'275109","ondisk_log_start":"140'275109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3525366029,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":282133,"num_read_kb":1092064,"num_write":276351,"num_write_kb":1105190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"143'289437","reported_seq":577412,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:40.942197+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:12:40.942197+0000","last_peered":"2023-10-14T01:12:40.942197+0000","last_clean":"2023-10-14T01:12:40.942197+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:12:40.942197+0000","last_undegraded":"2023-10-14T01:12:40.942197+0000","last_fullsized":"2023-10-14T01:12:40.942197+0000","mapping_epoch":13,"log_start":"140'279411","ondisk_log_start":"140'279411","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3573911801,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":289248,"num_read_kb":1114344,"num_write":278765,"num_write_kb":1114786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"143'284036","reported_seq":561945,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:43.625177+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:12:43.625177+0000","last_peered":"2023-10-14T01:12:43.625177+0000","last_clean":"2023-10-14T01:12:43.625177+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:12:43.625177+0000","last_undegraded":"2023-10-14T01:12:43.625177+0000","last_fullsized":"2023-10-14T01:12:43.625177+0000","mapping_epoch":13,"log_start":"140'274007","ondisk_log_start":"140'274007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":420,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 142s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3512099000,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":278692,"num_read_kb":1088763,"num_write":274749,"num_write_kb":1098883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":351,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:47.277469+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:11:47.277469+0000","last_peered":"2023-10-14T01:11:47.277469+0000","last_clean":"2023-10-14T01:11:47.277469+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:11:47.277469+0000","last_undegraded":"2023-10-14T01:11:47.277469+0000","last_fullsized":"2023-10-14T01:11:47.277469+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28266887480,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2253580,"num_read_kb":8765718,"num_write":2210224,"num_write_kb":8839425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18100568064,"data_stored":22412046960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80375,"ondisk_log_size":80375,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739286,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4955176,"kb_used_data":3319584,"kb_used_omap":214,"kb_used_meta":1635369,"kb_avail":89416664,"statfs":{"total":96636764160,"available":91562663936,"internally_reserved":0,"allocated":3399254016,"data_stored":4208412239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1674618468},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.63800000000000001,"15min":0.747},"min":{"1min":0.20100000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.1150000000000002,"5min":5.5860000000000003,"15min":7.492},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91900000000000004,"15min":0.90900000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":4.4619999999999997,"5min":6.6539999999999999,"15min":10.847},"last":0.29699999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:12:14 2023","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.27,"15min":1.3109999999999999},"min":{"1min":0.217,"5min":0.193,"15min":0.191},"max":{"1min":11.077999999999999,"5min":11.077999999999999,"15min":14.103999999999999},"last":5.0739999999999998},{"interface":"front","average":{"1min":2.044,"5min":1.5289999999999999,"15min":1.1870000000000001},"min":{"1min":0.249,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":14.586,"5min":14.586,"15min":14.586},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739274,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10662428,"kb_used_data":8842060,"kb_used_omap":689,"kb_used_meta":1819662,"kb_avail":83709412,"statfs":{"total":96636764160,"available":85718437888,"internally_reserved":0,"allocated":9054269440,"data_stored":11209727721,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":706241,"internal_metadata":1863334207},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,9,7,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.68200000000000005,"15min":0.69299999999999995},"min":{"1min":0.27900000000000003,"5min":0.20200000000000001,"15min":0.16800000000000001},"max":{"1min":1.629,"5min":10.552,"15min":10.891},"last":3.1659999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.192},"max":{"1min":1.681,"5min":10.52,"15min":10.856},"last":3.3999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:11:47 2023","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.1759999999999999,"15min":1.226},"min":{"1min":0.27400000000000002,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":8.7780000000000005,"5min":8.9469999999999992,"15min":43.871000000000002},"last":3.6110000000000002},{"interface":"front","average":{"1min":2.2250000000000001,"5min":1.3149999999999999,"15min":1.24},"min":{"1min":0.255,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":8.6489999999999991,"5min":8.6489999999999991,"15min":39.412999999999997},"last":3.5630000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739284,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7394748,"kb_used_data":5518004,"kb_used_omap":373,"kb_used_meta":1876362,"kb_avail":86977092,"statfs":{"total":96636764160,"available":89064542208,"internally_reserved":0,"allocated":5650436096,"data_stored":6995921963,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":1921395414},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:27 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":0.84299999999999997,"15min":0.90100000000000002},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.952,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":4.649},{"interface":"front","average":{"1min":0.497,"5min":0.63800000000000001,"15min":0.60099999999999998},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.17399999999999999},"max":{"1min":3.4489999999999998,"5min":9.2439999999999998,"15min":9.2439999999999998},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.96199999999999997,"15min":1.286},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":3.097,"5min":9.3460000000000001,"15min":10.539999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.93000000000000005,"15min":0.77500000000000002},"min":{"1min":0.253,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":1.8049999999999999,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5649063936,"data_stored":6994830970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9053556736,"data_stored":11209296184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3397947392,"data_stored":4207919806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:12:44.372 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:44.373 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:44.373 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:44.373 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:44.374 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:44.374 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:44.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:44.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:12:44.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:44.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:12:44.632 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:12:44.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:12:44.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:44.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:44.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:44.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:44.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:44.932+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:44.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:44.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:46.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:46.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:46.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:46.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:46.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:46.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:53.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:53.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:53.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:53.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:53.924 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:53.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:12:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:53.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:12:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:53.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:53.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:53.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:55.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:55.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:12:55.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:55.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:55.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:55.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:55.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:55.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:55.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:55.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:12:55.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:55.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:57.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:57.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:12:57.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:57.604+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:12:57.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:57.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:57.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:57.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:12:58.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:58.004+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:12:58.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:12:58.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:01.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:01.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:01.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:01.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:01.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:02.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:02.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:02.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:02.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:02.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:02.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:02.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:02.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:02.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:02.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:02.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:03.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:03.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:03.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:03.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:03.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:03.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:03.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:03.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:04.375 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:05.339 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:13:05.340 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:13:05.413 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2448,"stamp":"2023-10-14T01:13:03.665649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28295199576,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2260224,"num_read_kb":8791811,"num_write":2217103,"num_write_kb":8868813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80510,"ondisk_log_size":80510,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23272484,"kb_used_data":17734276,"kb_used_omap":1277,"kb_used_meta":5536898,"kb_avail":259843036,"statfs":{"total":289910292480,"available":266079268864,"internally_reserved":0,"allocated":18159898624,"data_stored":22470000995,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1308039,"internal_metadata":5669784185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,7,11,12,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16609280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3832,"num_read_kb":15264,"num_write":4003,"num_write_kb":16012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008801"},"pg_stats":[{"pgid":"2.7","version":"143'297355","reported_seq":584110,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:59.005962+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:12:59.005962+0000","last_peered":"2023-10-14T01:12:59.005962+0000","last_clean":"2023-10-14T01:12:59.005962+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:12:59.005962+0000","last_undegraded":"2023-10-14T01:12:59.005962+0000","last_fullsized":"2023-10-14T01:12:59.005962+0000","mapping_epoch":13,"log_start":"141'287302","ondisk_log_start":"141'287302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3578622390,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":288098,"num_read_kb":1115075,"num_write":279969,"num_write_kb":1119634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"143'297441","reported_seq":584800,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:58.991668+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:12:58.991668+0000","last_peered":"2023-10-14T01:12:58.991668+0000","last_clean":"2023-10-14T01:12:58.991668+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:12:58.991668+0000","last_undegraded":"2023-10-14T01:12:58.991668+0000","last_fullsized":"2023-10-14T01:12:58.991668+0000","mapping_epoch":13,"log_start":"141'287410","ondisk_log_start":"141'287410","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":300,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 150s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3571278192,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":290233,"num_read_kb":1113581,"num_write":279925,"num_write_kb":1119327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"143'268886","reported_seq":535682,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:58.925933+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:12:58.925933+0000","last_peered":"2023-10-14T01:12:58.925933+0000","last_clean":"2023-10-14T01:12:58.925933+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:12:58.925933+0000","last_undegraded":"2023-10-14T01:12:58.925933+0000","last_fullsized":"2023-10-14T01:12:58.925933+0000","mapping_epoch":13,"log_start":"140'258805","ondisk_log_start":"140'258805","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3417031058,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":267222,"num_read_kb":1053148,"num_write":268268,"num_write_kb":1073042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"143'262565","reported_seq":525943,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:58.914276+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:12:58.914276+0000","last_peered":"2023-10-14T01:12:58.914276+0000","last_clean":"2023-10-14T01:12:58.914276+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:12:58.914276+0000","last_undegraded":"2023-10-14T01:12:58.914276+0000","last_fullsized":"2023-10-14T01:12:58.914276+0000","mapping_epoch":13,"log_start":"140'252504","ondisk_log_start":"140'252504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":80,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 29s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3343876227,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":263830,"num_read_kb":1037797,"num_write":261917,"num_write_kb":1047590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"143'296638","reported_seq":594222,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:12:58.832408+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:12:58.832408+0000","last_peered":"2023-10-14T01:12:58.832408+0000","last_clean":"2023-10-14T01:12:58.832408+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:12:58.832408+0000","last_undegraded":"2023-10-14T01:12:58.832408+0000","last_fullsized":"2023-10-14T01:12:58.832408+0000","mapping_epoch":13,"log_start":"141'286609","ondisk_log_start":"141'286609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3761164607,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":298036,"num_read_kb":1166530,"num_write":294258,"num_write_kb":1176885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"143'286153","reported_seq":568156,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:13:01.279888+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:13:01.279888+0000","last_peered":"2023-10-14T01:13:01.279888+0000","last_clean":"2023-10-14T01:13:01.279888+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:13:01.279888+0000","last_undegraded":"2023-10-14T01:13:01.279888+0000","last_fullsized":"2023-10-14T01:13:01.279888+0000","mapping_epoch":13,"log_start":"141'276109","ondisk_log_start":"141'276109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3529310477,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":283068,"num_read_kb":1095792,"num_write":277307,"num_write_kb":1109014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"143'290459","reported_seq":579422,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:13:01.219347+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:13:01.219347+0000","last_peered":"2023-10-14T01:13:01.219347+0000","last_clean":"2023-10-14T01:13:01.219347+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:13:01.219347+0000","last_undegraded":"2023-10-14T01:13:01.219347+0000","last_fullsized":"2023-10-14T01:13:01.219347+0000","mapping_epoch":13,"log_start":"141'280411","ondisk_log_start":"141'280411","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3578048761,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":290218,"num_read_kb":1118216,"num_write":279772,"num_write_kb":1118814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"143'284843","reported_seq":563467,"reported_epoch":143,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:58.972304+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:12:58.972304+0000","last_peered":"2023-10-14T01:12:58.972304+0000","last_clean":"2023-10-14T01:12:58.972304+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:12:58.972304+0000","last_undegraded":"2023-10-14T01:12:58.972304+0000","last_fullsized":"2023-10-14T01:12:58.972304+0000","mapping_epoch":13,"log_start":"141'274807","ondisk_log_start":"141'274807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":458,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 157s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3515277496,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":279393,"num_read_kb":1091563,"num_write":275518,"num_write_kb":1101959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":351,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-10-14T01:11:47.277469+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:11:47.277469+0000","last_peered":"2023-10-14T01:11:47.277469+0000","last_clean":"2023-10-14T01:11:47.277469+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:11:47.277469+0000","last_undegraded":"2023-10-14T01:11:47.277469+0000","last_fullsized":"2023-10-14T01:11:47.277469+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28294609208,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2260098,"num_read_kb":8791702,"num_write":2216934,"num_write_kb":8866265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18157498368,"data_stored":22468977264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80383,"ondisk_log_size":80383,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739290,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5015476,"kb_used_data":3331308,"kb_used_omap":214,"kb_used_meta":1683945,"kb_avail":89356364,"statfs":{"total":96636764160,"available":91500916736,"internally_reserved":0,"allocated":3411259392,"data_stored":4220417615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1724360292},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.63800000000000001,"15min":0.747},"min":{"1min":0.20100000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":2.1150000000000002,"5min":5.5860000000000003,"15min":7.492},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91900000000000004,"15min":0.90900000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":4.4619999999999997,"5min":6.6539999999999999,"15min":10.847},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:12:14 2023","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.27,"15min":1.3109999999999999},"min":{"1min":0.217,"5min":0.193,"15min":0.191},"max":{"1min":11.077999999999999,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":2.044,"5min":1.5289999999999999,"15min":1.1870000000000001},"min":{"1min":0.249,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":14.586,"5min":14.586,"15min":14.586},"last":0.42799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739277,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10740000,"kb_used_data":8865496,"kb_used_omap":689,"kb_used_meta":1873806,"kb_avail":83631840,"statfs":{"total":96636764160,"available":85639004160,"internally_reserved":0,"allocated":9078267904,"data_stored":11233726185,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":706241,"internal_metadata":1918777663},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,5,7,10,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.68200000000000005,"15min":0.69299999999999995},"min":{"1min":0.27900000000000003,"5min":0.20200000000000001,"15min":0.16800000000000001},"max":{"1min":1.629,"5min":10.552,"15min":10.891},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.192},"max":{"1min":1.681,"5min":10.52,"15min":10.856},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:12:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.1819999999999999,"15min":1.089},"min":{"1min":0.25,"5min":0.246,"15min":0.222},"max":{"1min":3.6110000000000002,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.4019999999999999,"15min":1.238},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.55400000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739288,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7517008,"kb_used_data":5537472,"kb_used_omap":373,"kb_used_meta":1979146,"kb_avail":86854832,"statfs":{"total":96636764160,"available":88939347968,"internally_reserved":0,"allocated":5670371328,"data_stored":7015857195,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":2026646230},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:27 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":0.84299999999999997,"15min":0.90100000000000002},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.952,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":1.153},{"interface":"front","average":{"1min":0.497,"5min":0.63800000000000001,"15min":0.60099999999999998},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.17399999999999999},"max":{"1min":3.4489999999999998,"5min":9.2439999999999998,"15min":9.2439999999999998},"last":0.81699999999999995}]},{"osd":2,"last update":"Sat Oct 14 01:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.96199999999999997,"15min":1.286},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":3.097,"5min":9.3460000000000001,"15min":10.539999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.93000000000000005,"15min":0.77500000000000002},"min":{"1min":0.253,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":1.8049999999999999,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5669236736,"data_stored":7015003770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9078112256,"data_stored":11233851704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3410149376,"data_stored":4220121790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:13:05.415 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:05.415 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:05.416 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:05.416 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:05.416 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:05.417 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:06.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:06.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:06.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:06.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:06.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:06.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:06.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:06.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:06.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:06.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:06.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:06.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:07.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:07.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:07.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:07.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:07.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:07.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:07.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:07.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:13:07.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:07.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:07.834 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:07.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:07.836 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:07.836+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:07.864 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:07.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:07.864 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:07.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:07.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:07.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:08.022 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:08.031 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:08.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:08.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.044+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:08.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:08.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:08.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:08.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:09.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:09.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:09.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:09.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:09.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:09.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:09.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:09.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:11.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:11.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:11.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:11.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:11.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:11.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:13:11.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:11.743 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:11.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:11.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:12.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:12.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:13:12.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:12.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:12.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:12.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:12.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:12.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:12.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:12.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:12.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:12.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:12.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:12.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:13.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:13.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:13.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:13.400 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:13.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:13.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:13.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:13.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:13.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:13.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:13.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:13.983 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:13.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:13.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:13.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:14.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:14.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:14.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:14.035 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:14.338 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:14.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:14.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:14.344+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:14.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:14.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:14.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:14.392+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:14.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:14.863 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:14.898 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:14.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:14.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:14.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.912+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:14.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:14.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:16.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:16.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:16.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:16.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:16.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:16.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:16.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:16.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:16.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:16.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:16.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:16.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:18.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:18.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:18.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:18.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:18.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:18.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:18.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:18.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:18.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:18.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:18.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:18.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:19.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:19.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:19.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:19.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.576+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:19.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:19.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:19.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:13:19.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:13:19.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:13:19.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.812+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:19.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:19.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:19.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:19.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:19.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:19.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:19.824+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:22.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:22.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:22.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:22.468+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:23.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:13:23.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.100+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:23.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.124+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:23.141 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.140+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:23.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.144+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:23.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.144+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:23.147 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.144+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:23.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:23.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:23.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:23.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:23.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:23.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:23.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:23.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:23.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:23.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:13:23.894 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:23.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:23.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:23.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:23.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:23.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:23.932+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:23.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:23.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:13:23.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:23.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:13:24.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:13:24.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:24.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:24.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:24.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:24.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:24.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:24.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:24.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:13:24.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:24.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:24.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:24.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:24.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:24.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:13:24.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:13:24.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:24.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:24.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:24.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:24.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:24.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:24.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:24.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:25.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:25.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:25.318 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:25.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:25.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:25.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:25.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:25.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:25.319 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:25.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:25.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:25.316+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:25.418 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:25.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:25.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:26.382 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:13:26.383 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:13:26.460 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2460,"stamp":"2023-10-14T01:13:25.669965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28334553944,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2269849,"num_read_kb":8830199,"num_write":2226690,"num_write_kb":8907161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80450,"ondisk_log_size":80450,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23626544,"kb_used_data":17811864,"kb_used_omap":1277,"kb_used_meta":5813378,"kb_avail":259488976,"statfs":{"total":289910292480,"available":265716711424,"internally_reserved":0,"allocated":18239348736,"data_stored":22549427509,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1308039,"internal_metadata":5952899705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,10,35,13,12,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22921216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5500,"num_read_kb":21936,"num_write":5578,"num_write_kb":22312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002666"},"pg_stats":[{"pgid":"2.7","version":"144'298754","reported_seq":586808,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:24.803297+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:13:24.803297+0000","last_peered":"2023-10-14T01:13:24.803297+0000","last_clean":"2023-10-14T01:13:24.803297+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:13:24.803297+0000","last_undegraded":"2023-10-14T01:13:24.803297+0000","last_fullsized":"2023-10-14T01:13:24.803297+0000","mapping_epoch":13,"log_start":"141'288702","ondisk_log_start":"141'288702","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3584008630,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":289371,"num_read_kb":1120151,"num_write":281279,"num_write_kb":1124874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'298853","reported_seq":587515,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:24.791420+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:13:24.791420+0000","last_peered":"2023-10-14T01:13:24.791420+0000","last_clean":"2023-10-14T01:13:24.791420+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:13:24.791420+0000","last_undegraded":"2023-10-14T01:13:24.791420+0000","last_fullsized":"2023-10-14T01:13:24.791420+0000","mapping_epoch":13,"log_start":"141'288810","ondisk_log_start":"141'288810","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":338,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 176s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3576623472,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":291509,"num_read_kb":1118665,"num_write":281226,"num_write_kb":1124531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'270109","reported_seq":538153,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:24.791896+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:13:24.791896+0000","last_peered":"2023-10-14T01:13:24.791896+0000","last_clean":"2023-10-14T01:13:24.791896+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:13:24.791896+0000","last_undegraded":"2023-10-14T01:13:24.791896+0000","last_fullsized":"2023-10-14T01:13:24.791896+0000","mapping_epoch":13,"log_start":"141'260105","ondisk_log_start":"141'260105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3422085522,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":268469,"num_read_kb":1058132,"num_write":269491,"num_write_kb":1077934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'263770","reported_seq":528356,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:24.788775+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:13:24.788775+0000","last_peered":"2023-10-14T01:13:24.788775+0000","last_clean":"2023-10-14T01:13:24.788775+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:13:24.788775+0000","last_undegraded":"2023-10-14T01:13:24.788775+0000","last_fullsized":"2023-10-14T01:13:24.788775+0000","mapping_epoch":13,"log_start":"141'253704","ondisk_log_start":"141'253704","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":143,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 55s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3348701315,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":265024,"num_read_kb":1042565,"num_write":263087,"num_write_kb":1052270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"144'297902","reported_seq":596836,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:24.021324+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:13:24.021324+0000","last_peered":"2023-10-14T01:13:24.021324+0000","last_clean":"2023-10-14T01:13:24.021324+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:13:24.021324+0000","last_undegraded":"2023-10-14T01:13:24.021324+0000","last_fullsized":"2023-10-14T01:13:24.021324+0000","mapping_epoch":13,"log_start":"141'287809","ondisk_log_start":"141'287809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3766374719,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":299378,"num_read_kb":1171878,"num_write":295520,"num_write_kb":1181933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"144'287216","reported_seq":570208,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:21.559937+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:13:21.559937+0000","last_peered":"2023-10-14T01:13:21.559937+0000","last_clean":"2023-10-14T01:13:21.559937+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:13:21.559937+0000","last_undegraded":"2023-10-14T01:13:21.559937+0000","last_fullsized":"2023-10-14T01:13:21.559937+0000","mapping_epoch":13,"log_start":"141'277209","ondisk_log_start":"141'277209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3533545741,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":284040,"num_read_kb":1099664,"num_write":278349,"num_write_kb":1113182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"144'291541","reported_seq":581558,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:21.561328+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:13:21.561328+0000","last_peered":"2023-10-14T01:13:21.561328+0000","last_clean":"2023-10-14T01:13:21.561328+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:13:21.561328+0000","last_undegraded":"2023-10-14T01:13:21.561328+0000","last_fullsized":"2023-10-14T01:13:21.561328+0000","mapping_epoch":13,"log_start":"141'281511","ondisk_log_start":"141'281511","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3582374137,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":291254,"num_read_kb":1122352,"num_write":280829,"num_write_kb":1123042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"144'286135","reported_seq":566064,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:24.792175+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:13:24.792175+0000","last_peered":"2023-10-14T01:13:24.792175+0000","last_clean":"2023-10-14T01:13:24.792175+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:13:24.792175+0000","last_undegraded":"2023-10-14T01:13:24.792175+0000","last_fullsized":"2023-10-14T01:13:24.792175+0000","mapping_epoch":13,"log_start":"141'276107","ondisk_log_start":"141'276107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":506,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 183s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3520250040,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":280678,"num_read_kb":1096683,"num_write":276740,"num_write_kb":1106847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":352,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:11:54.587133+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:11:54.587133+0000","last_peered":"2023-10-14T01:11:54.587133+0000","last_clean":"2023-10-14T01:11:54.587133+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:11:54.587133+0000","last_undegraded":"2023-10-14T01:11:54.587133+0000","last_fullsized":"2023-10-14T01:11:54.587133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28333963576,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2269723,"num_read_kb":8830090,"num_write":2226521,"num_write_kb":8904613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18236829696,"data_stored":22548308592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80323,"ondisk_log_size":80323,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739294,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5059512,"kb_used_data":3343408,"kb_used_omap":214,"kb_used_meta":1715881,"kb_avail":89312328,"statfs":{"total":96636764160,"available":91455823872,"internally_reserved":0,"allocated":3423649792,"data_stored":4232800149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1757062756},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:14 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.58899999999999997,"15min":0.73399999999999999},"min":{"1min":0.19900000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.2679999999999998,"5min":5.2679999999999998,"15min":7.492},"last":0.182},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.755,"15min":0.873},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.1080000000000001,"5min":6.2080000000000002,"15min":10.847},"last":0.214}]},{"osd":1,"last update":"Sat Oct 14 01:13:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.375,"15min":1.341},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.191},"max":{"1min":5.7220000000000004,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.248},{"interface":"front","average":{"1min":1.667,"5min":1.657,"15min":1.264},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":9.1920000000000002,"5min":14.586,"15min":14.586},"last":0.36399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739282,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10949216,"kb_used_data":8906520,"kb_used_omap":689,"kb_used_meta":2041998,"kb_avail":83422624,"statfs":{"total":96636764160,"available":85424766976,"internally_reserved":0,"allocated":9120276480,"data_stored":11275726895,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":706241,"internal_metadata":2091006271},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,32,5,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.68200000000000005,"15min":0.69299999999999995},"min":{"1min":0.27900000000000003,"5min":0.20200000000000001,"15min":0.16800000000000001},"max":{"1min":1.629,"5min":10.552,"15min":10.891},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.27100000000000002,"5min":0.23899999999999999,"15min":0.192},"max":{"1min":1.681,"5min":10.52,"15min":10.856},"last":0.373}]},{"osd":2,"last update":"Sat Oct 14 01:12:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.1819999999999999,"15min":1.089},"min":{"1min":0.25,"5min":0.246,"15min":0.222},"max":{"1min":3.6110000000000002,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.4019999999999999,"15min":1.238},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.35499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739293,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7617816,"kb_used_data":5561936,"kb_used_omap":373,"kb_used_meta":2055498,"kb_avail":86754024,"statfs":{"total":96636764160,"available":88836120576,"internally_reserved":0,"allocated":5695422464,"data_stored":7040900465,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":382250,"internal_metadata":2104830678},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,6,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:27 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":0.84299999999999997,"15min":0.90100000000000002},"min":{"1min":0.19800000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":6.952,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.63800000000000001,"15min":0.60099999999999998},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.17399999999999999},"max":{"1min":3.4489999999999998,"5min":9.2439999999999998,"15min":9.2439999999999998},"last":0.23100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:12:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.96199999999999997,"15min":1.286},"min":{"1min":0.246,"5min":0.20699999999999999,"15min":0.17599999999999999},"max":{"1min":3.097,"5min":9.3460000000000001,"15min":10.539999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.93000000000000005,"15min":0.77500000000000002},"min":{"1min":0.253,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":1.8049999999999999,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":3.2109999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5694398464,"data_stored":7040165498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9119801344,"data_stored":11275540792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3422629888,"data_stored":4232602302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:13:26.461 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:26.462 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:26.462 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:26.463 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:26.463 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:26.464 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:26.574 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:26.571+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:26.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:26.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:26.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:26.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:26.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:26.607+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:26.705 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:26.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:26.706 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:26.703+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:28.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:28.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:28.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:28.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:28.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:28.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:28.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:28.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:28.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:28.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:29.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:29.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:29.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:29.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:29.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:29.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:29.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:29.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:30.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:30.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:30.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:30.207+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:30.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:30.219+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:30.243 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:30.239+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:30.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:30.239+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:30.244 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:30.239+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:30.559 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:13:30.555+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:13:31.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:31.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:31.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:31.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:32.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:32.440 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:32.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:32.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:32.441 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:32.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:33.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:33.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:33.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:33.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:33.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:33.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:34.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:34.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:13:34.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:34.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:34.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:34.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:34.819 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:34.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:34.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:34.827+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:34.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:34.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:34.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:34.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:35.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:35.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:35.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:35.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:35.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:35.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:35.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:35.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:35.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:35.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:35.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:35.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:36.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:36.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:36.136 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:36.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:36.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:36.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:36.395+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:36.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:36.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:36.408 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:36.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:36.408 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:36.403+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:36.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:36.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:36.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:36.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.547+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:36.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:36.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:36.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:38.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:38.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:38.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:38.335+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:38.349 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:38.347+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:38.349 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:38.347+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:40.131 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:40.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:40.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:40.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:40.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:40.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:40.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:40.171+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:40.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:40.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:40.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:40.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:40.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:40.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:42.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:42.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:42.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:42.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:42.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:42.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:42.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:42.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:42.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:42.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:42.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:42.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:43.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:43.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:43.726 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:43.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:43.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:43.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:44.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:44.743+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:45.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:45.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:13:45.344 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:45.339+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:45.392 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:45.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:45.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:45.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:45.423 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:45.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:46.465 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:46.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:46.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:46.785 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:46.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:46.798 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:46.795+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:46.806 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:46.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:46.807 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:46.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:46.807 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:46.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:47.204 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:13:47.205 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:13:47.258 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2470,"stamp":"2023-10-14T01:13:45.674881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28367915864,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2277920,"num_read_kb":8862371,"num_write":2234766,"num_write_kb":8939465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80548,"ondisk_log_size":80548,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23849164,"kb_used_data":17876784,"kb_used_omap":1279,"kb_used_meta":5971072,"kb_avail":259266356,"statfs":{"total":289910292480,"available":265488748544,"internally_reserved":0,"allocated":18305826816,"data_stored":22615905589,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1309719,"internal_metadata":6114378729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,4,6,12,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22962176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5542,"num_read_kb":22092,"num_write":5555,"num_write_kb":22220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003570"},"pg_stats":[{"pgid":"2.7","version":"144'299867","reported_seq":589011,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:45.388812+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:13:45.388812+0000","last_peered":"2023-10-14T01:13:45.388812+0000","last_clean":"2023-10-14T01:13:45.388812+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:13:45.388812+0000","last_undegraded":"2023-10-14T01:13:45.388812+0000","last_fullsized":"2023-10-14T01:13:45.388812+0000","mapping_epoch":13,"log_start":"141'289802","ondisk_log_start":"141'289802","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3588325814,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":290452,"num_read_kb":1124459,"num_write":282324,"num_write_kb":1129054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'299936","reported_seq":589611,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:45.363520+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:13:45.363520+0000","last_peered":"2023-10-14T01:13:45.363520+0000","last_clean":"2023-10-14T01:13:45.363520+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:13:45.363520+0000","last_undegraded":"2023-10-14T01:13:45.363520+0000","last_fullsized":"2023-10-14T01:13:45.363520+0000","mapping_epoch":13,"log_start":"141'289910","ondisk_log_start":"141'289910","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":372,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 197s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3580752240,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":292502,"num_read_kb":1122633,"num_write":282223,"num_write_kb":1128519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'271168","reported_seq":540206,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:45.338518+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:13:45.338518+0000","last_peered":"2023-10-14T01:13:45.338518+0000","last_clean":"2023-10-14T01:13:45.338518+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:13:45.338518+0000","last_undegraded":"2023-10-14T01:13:45.338518+0000","last_fullsized":"2023-10-14T01:13:45.338518+0000","mapping_epoch":13,"log_start":"141'261105","ondisk_log_start":"141'261105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3426537874,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":269464,"num_read_kb":1062092,"num_write":270550,"num_write_kb":1082170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'264779","reported_seq":530273,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:45.322583+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:13:45.322583+0000","last_peered":"2023-10-14T01:13:45.322583+0000","last_clean":"2023-10-14T01:13:45.322583+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:13:45.322583+0000","last_undegraded":"2023-10-14T01:13:45.322583+0000","last_fullsized":"2023-10-14T01:13:45.322583+0000","mapping_epoch":13,"log_start":"141'254705","ondisk_log_start":"141'254705","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":201,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 76s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3352694915,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":265922,"num_read_kb":1046137,"num_write":264063,"num_write_kb":1056174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"144'298954","reported_seq":598957,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:44.133828+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:13:44.133828+0000","last_peered":"2023-10-14T01:13:44.133828+0000","last_clean":"2023-10-14T01:13:44.133828+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:13:44.133828+0000","last_undegraded":"2023-10-14T01:13:44.133828+0000","last_fullsized":"2023-10-14T01:13:44.133828+0000","mapping_epoch":13,"log_start":"141'288909","ondisk_log_start":"141'288909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3770646847,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":300444,"num_read_kb":1176130,"num_write":296567,"num_write_kb":1186121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"144'288215","reported_seq":572193,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.869186+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:13:41.869186+0000","last_peered":"2023-10-14T01:13:41.869186+0000","last_clean":"2023-10-14T01:13:41.869186+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:13:41.869186+0000","last_undegraded":"2023-10-14T01:13:41.869186+0000","last_fullsized":"2023-10-14T01:13:41.869186+0000","mapping_epoch":13,"log_start":"141'278209","ondisk_log_start":"141'278209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3537522957,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":285008,"num_read_kb":1103528,"num_write":279316,"num_write_kb":1117050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"144'292597","reported_seq":583661,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.872768+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:13:41.872768+0000","last_peered":"2023-10-14T01:13:41.872768+0000","last_clean":"2023-10-14T01:13:41.872768+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:13:41.872768+0000","last_undegraded":"2023-10-14T01:13:41.872768+0000","last_fullsized":"2023-10-14T01:13:41.872768+0000","mapping_epoch":13,"log_start":"141'282511","ondisk_log_start":"141'282511","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3586572537,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":292276,"num_read_kb":1126432,"num_write":281843,"num_write_kb":1127098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"144'287163","reported_seq":568157,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:45.333841+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:13:45.333841+0000","last_peered":"2023-10-14T01:13:45.333841+0000","last_clean":"2023-10-14T01:13:45.333841+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:13:45.333841+0000","last_undegraded":"2023-10-14T01:13:45.333841+0000","last_fullsized":"2023-10-14T01:13:45.333841+0000","mapping_epoch":13,"log_start":"141'277107","ondisk_log_start":"141'277107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":548,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 204s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3524272312,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":281726,"num_read_kb":1100851,"num_write":277711,"num_write_kb":1110731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":352,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-10-14T01:11:54.587133+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:11:54.587133+0000","last_peered":"2023-10-14T01:11:54.587133+0000","last_clean":"2023-10-14T01:11:54.587133+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:11:54.587133+0000","last_undegraded":"2023-10-14T01:11:54.587133+0000","last_fullsized":"2023-10-14T01:11:54.587133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28367325496,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2277794,"num_read_kb":8862262,"num_write":2234597,"num_write_kb":8936917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18302615552,"data_stored":22614094448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80421,"ondisk_log_size":80421,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739298,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5139468,"kb_used_data":3355264,"kb_used_omap":214,"kb_used_meta":1783977,"kb_avail":89232372,"statfs":{"total":96636764160,"available":91373948928,"internally_reserved":0,"allocated":3435790336,"data_stored":4244940693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1826793060},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:14 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.58899999999999997,"15min":0.73399999999999999},"min":{"1min":0.19900000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.2679999999999998,"5min":5.2679999999999998,"15min":7.492},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.755,"15min":0.873},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.1080000000000001,"5min":6.2080000000000002,"15min":10.847},"last":0.27100000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:13:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.375,"15min":1.341},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.191},"max":{"1min":5.7220000000000004,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.433},{"interface":"front","average":{"1min":1.667,"5min":1.657,"15min":1.264},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":9.1920000000000002,"5min":14.586,"15min":14.586},"last":0.45100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739286,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11093784,"kb_used_data":8939536,"kb_used_omap":689,"kb_used_meta":2153550,"kb_avail":83278056,"statfs":{"total":96636764160,"available":85276729344,"internally_reserved":0,"allocated":9154084864,"data_stored":11309535279,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":706241,"internal_metadata":2205235519},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2,3,5,6,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.73299999999999998,"15min":0.67800000000000005},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16800000000000001},"max":{"1min":3.1659999999999999,"5min":10.552,"15min":10.891},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.73499999999999999,"15min":0.69499999999999995},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":3.3999999999999999,"5min":10.52,"15min":10.856},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:12:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":1.1819999999999999,"15min":1.089},"min":{"1min":0.25,"5min":0.246,"15min":0.222},"max":{"1min":3.6110000000000002,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.4019999999999999,"15min":1.238},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.20200000000000001},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.79800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739297,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7615912,"kb_used_data":5581984,"kb_used_omap":374,"kb_used_meta":2033545,"kb_avail":86755928,"statfs":{"total":96636764160,"available":88838070272,"internally_reserved":0,"allocated":5715951616,"data_stored":7061429617,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":383930,"internal_metadata":2082350150},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:31 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":0.92400000000000004,"15min":0.91600000000000004},"min":{"1min":0.186,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.258,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.57199999999999995,"15min":0.58999999999999997},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.17399999999999999},"max":{"1min":3.504,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:13:31 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.873,"15min":1.262},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":9.5809999999999995,"5min":9.5809999999999995,"15min":10.539999999999999},"last":2.2490000000000001},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.006,"15min":0.79700000000000004},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":3.5,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":1.4099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5714849792,"data_stored":7060616826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9153003520,"data_stored":11308742968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3434762240,"data_stored":4244734654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:13:47.259 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:47.260 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:47.260 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:47.260 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:47.261 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:47.261 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:50.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:50.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:50.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:50.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:51.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:51.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:13:51.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:51.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:51.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:51.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:51.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:51.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:51.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:51.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:51.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:51.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:51.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:51.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:51.565 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:51.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:13:51.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:51.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:13:51.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:51.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:51.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:51.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:51.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:51.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:51.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:51.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:53.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:53.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:53.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:53.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:53.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:53.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:53.307 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:53.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:53.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:53.343+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:53.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:53.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:53.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:53.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:55.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:55.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:13:55.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:55.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:55.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:55.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:55.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:55.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:56.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:56.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:56.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:56.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:56.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:56.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:57.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:57.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:13:57.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:57.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:57.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:57.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:57.081 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:57.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:13:57.528 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:57.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:13:57.528 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:57.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:57.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:57.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:57.529 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:57.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:57.926 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:57.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:57.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:57.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:57.946 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:57.943+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:58.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:58.067+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:13:58.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:58.075+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:13:58.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:58.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:13:58.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:58.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:13:58.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:58.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:13:58.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:13:58.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:13:58.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:58.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:13:58.870 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:58.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:13:58.871 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:13:58.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:04.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:04.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:04.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:04.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:04.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:04.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:05.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:05.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:05.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:05.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:05.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:05.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:05.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:05.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:05.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:05.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:06.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:06.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.511+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:06.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:06.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:14:06.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:06.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:06.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:06.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:06.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:06.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:06.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:06.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:07.262 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:08.104 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:14:08.104 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:14:08.134 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2481,"stamp":"2023-10-14T01:14:07.685976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28402289496,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2286216,"num_read_kb":8895483,"num_write":2243106,"num_write_kb":8972825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80561,"ondisk_log_size":80561,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24019116,"kb_used_data":17941076,"kb_used_omap":1279,"kb_used_meta":6076736,"kb_avail":259096404,"statfs":{"total":289910292480,"available":265314717696,"internally_reserved":0,"allocated":18371661824,"data_stored":22681740597,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1309719,"internal_metadata":6222578665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,6,5,6,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22949888,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5503,"num_read_kb":21968,"num_write":5568,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005017"},"pg_stats":[{"pgid":"2.7","version":"144'300905","reported_seq":591102,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:14:05.983360+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:14:05.983360+0000","last_peered":"2023-10-14T01:14:05.983360+0000","last_clean":"2023-10-14T01:14:05.983360+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:14:05.983360+0000","last_undegraded":"2023-10-14T01:14:05.983360+0000","last_fullsized":"2023-10-14T01:14:05.983360+0000","mapping_epoch":13,"log_start":"142'290902","ondisk_log_start":"142'290902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3592536502,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":291487,"num_read_kb":1128595,"num_write":283322,"num_write_kb":1133046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"144'300991","reported_seq":591693,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:05.994603+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:14:05.994603+0000","last_peered":"2023-10-14T01:14:05.994603+0000","last_clean":"2023-10-14T01:14:05.994603+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:14:05.994603+0000","last_undegraded":"2023-10-14T01:14:05.994603+0000","last_fullsized":"2023-10-14T01:14:05.994603+0000","mapping_epoch":13,"log_start":"142'290910","ondisk_log_start":"142'290910","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":415,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 217s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3584790896,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":293503,"num_read_kb":1126637,"num_write":283212,"num_write_kb":1132475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"144'272121","reported_seq":542105,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:14:05.977726+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:14:05.977726+0000","last_peered":"2023-10-14T01:14:05.977726+0000","last_clean":"2023-10-14T01:14:05.977726+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:14:05.977726+0000","last_undegraded":"2023-10-14T01:14:05.977726+0000","last_fullsized":"2023-10-14T01:14:05.977726+0000","mapping_epoch":13,"log_start":"141'262105","ondisk_log_start":"141'262105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3430449554,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":270410,"num_read_kb":1065860,"num_write":271503,"num_write_kb":1085982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"144'265803","reported_seq":532250,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:05.967140+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:14:05.967140+0000","last_peered":"2023-10-14T01:14:05.967140+0000","last_clean":"2023-10-14T01:14:05.967140+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:14:05.967140+0000","last_undegraded":"2023-10-14T01:14:05.967140+0000","last_fullsized":"2023-10-14T01:14:05.967140+0000","mapping_epoch":13,"log_start":"141'255705","ondisk_log_start":"141'255705","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":257,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 96s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3356762243,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":266868,"num_read_kb":1049917,"num_write":265052,"num_write_kb":1060130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"144'299977","reported_seq":601015,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:14:04.454101+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:14:04.454101+0000","last_peered":"2023-10-14T01:14:04.454101+0000","last_clean":"2023-10-14T01:14:04.454101+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:14:04.454101+0000","last_undegraded":"2023-10-14T01:14:04.454101+0000","last_fullsized":"2023-10-14T01:14:04.454101+0000","mapping_epoch":13,"log_start":"142'289909","ondisk_log_start":"142'289909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3774869823,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":301477,"num_read_kb":1180234,"num_write":297587,"num_write_kb":1190201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"144'289443","reported_seq":574640,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:14:07.151814+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:14:07.151814+0000","last_peered":"2023-10-14T01:14:07.151814+0000","last_clean":"2023-10-14T01:14:07.151814+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:14:07.151814+0000","last_undegraded":"2023-10-14T01:14:07.151814+0000","last_fullsized":"2023-10-14T01:14:07.151814+0000","mapping_epoch":13,"log_start":"142'279409","ondisk_log_start":"142'279409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3542446349,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":286207,"num_read_kb":1108316,"num_write":280508,"num_write_kb":1121818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"144'293877","reported_seq":586165,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:14:07.179861+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:14:07.179861+0000","last_peered":"2023-10-14T01:14:07.179861+0000","last_clean":"2023-10-14T01:14:07.179861+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:14:07.179861+0000","last_undegraded":"2023-10-14T01:14:07.179861+0000","last_fullsized":"2023-10-14T01:14:07.179861+0000","mapping_epoch":13,"log_start":"142'283811","ondisk_log_start":"142'283811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3591639289,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":293478,"num_read_kb":1131236,"num_write":283078,"num_write_kb":1132038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"144'288175","reported_seq":570118,"reported_epoch":144,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:05.945850+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:14:05.945850+0000","last_peered":"2023-10-14T01:14:05.945850+0000","last_clean":"2023-10-14T01:14:05.945850+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:14:05.945850+0000","last_undegraded":"2023-10-14T01:14:05.945850+0000","last_fullsized":"2023-10-14T01:14:05.945850+0000","mapping_epoch":13,"log_start":"142'278107","ondisk_log_start":"142'278107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":598,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 224s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3528204472,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":282660,"num_read_kb":1104579,"num_write":278675,"num_write_kb":1114587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":353,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:09.655919+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:13:09.655919+0000","last_peered":"2023-10-14T01:13:09.655919+0000","last_clean":"2023-10-14T01:13:09.655919+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:13:09.655919+0000","last_undegraded":"2023-10-14T01:13:09.655919+0000","last_fullsized":"2023-10-14T01:13:09.655919+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28401699128,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2286090,"num_read_kb":8895374,"num_write":2242937,"num_write_kb":8970277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18370052096,"data_stored":22681530992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80434,"ondisk_log_size":80434,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739303,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5189972,"kb_used_data":3369676,"kb_used_omap":214,"kb_used_meta":1820073,"kb_avail":89181868,"statfs":{"total":96636764160,"available":91322232832,"internally_reserved":0,"allocated":3450548224,"data_stored":4259698581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1863755364},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:14 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.58899999999999997,"15min":0.73399999999999999},"min":{"1min":0.19900000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.2679999999999998,"5min":5.2679999999999998,"15min":7.492},"last":0.313},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.755,"15min":0.873},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.1080000000000001,"5min":6.2080000000000002,"15min":10.847},"last":0.47599999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:13:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.375,"15min":1.341},"min":{"1min":0.22600000000000001,"5min":0.193,"15min":0.191},"max":{"1min":5.7220000000000004,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.667,"5min":1.657,"15min":1.264},"min":{"1min":0.25900000000000001,"5min":0.20899999999999999,"15min":0.19},"max":{"1min":9.1920000000000002,"5min":14.586,"15min":14.586},"last":0.51500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739290,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11139788,"kb_used_data":8970116,"kb_used_omap":689,"kb_used_meta":2168974,"kb_avail":83232052,"statfs":{"total":96636764160,"available":85229621248,"internally_reserved":0,"allocated":9185398784,"data_stored":11340849199,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":706241,"internal_metadata":2221029695},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,1,5,4,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.73299999999999998,"15min":0.67800000000000005},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16800000000000001},"max":{"1min":3.1659999999999999,"5min":10.552,"15min":10.891},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.73499999999999999,"15min":0.69499999999999995},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":3.3999999999999999,"5min":10.52,"15min":10.856},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:13:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.2030000000000001,"15min":1.0600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":7.5540000000000003,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.6279999999999999,"15min":1.2350000000000001},"min":{"1min":0.23899999999999999,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":11.06,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.35699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739301,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7689356,"kb_used_data":5601284,"kb_used_omap":374,"kb_used_meta":2087689,"kb_avail":86682484,"statfs":{"total":96636764160,"available":88762863616,"internally_reserved":0,"allocated":5735714816,"data_stored":7081192817,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":383930,"internal_metadata":2137793606},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:31 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":0.92400000000000004,"15min":0.91600000000000004},"min":{"1min":0.186,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.258,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.57199999999999995,"15min":0.58999999999999997},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.17399999999999999},"max":{"1min":3.504,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:13:31 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.873,"15min":1.262},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":9.5809999999999995,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.006,"15min":0.79700000000000004},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":3.5,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.314}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5735026688,"data_stored":7080793722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9185517568,"data_stored":11341257016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3449507840,"data_stored":4259480254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:14:08.135 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:08.136 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:08.136 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:08.136 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:08.137 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:08.137 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:09.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:14:09.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:09.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:14:09.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:09.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:09.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:09.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:09.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:09.815+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:10.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:10.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:10.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:10.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:10.679 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:10.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:10.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:10.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:10.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:10.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:10.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:10.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:10.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:10.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:10.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:10.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:14:10.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:10.919+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:10.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:10.923+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:10.948 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:10.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:10.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:10.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:10.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:10.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:10.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:10.951+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:15.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:15.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:15.413 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:15.411+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:15.414 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:15.411+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:15.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:15.411+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:15.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:15.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:14:15.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:15.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:15.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:15.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:15.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:15.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:15.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:15.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:16.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:16.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.243+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:16.251 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:16.294 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.291+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:16.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:14:16.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:14:16.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:16.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:16.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:16.869 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:16.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:17.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:17.039+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:17.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:17.075+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:17.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:17.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:17.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:17.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:17.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:17.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:20.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:20.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:14:21.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:20.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:14:21.013 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:20.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:14:21.022 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.018+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:21.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:21.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.066+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:21.077 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:21.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:21.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:21.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:14:21.444 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.438+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:21.464 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:21.481 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:21.501 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:21.501 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:21.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:21.506+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:24.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:24.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:14:24.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:24.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:24.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:24.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:24.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:24.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:24.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:24.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:24.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:24.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:25.133 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:25.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:14:25.134 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:25.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:25.152 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:25.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:25.154 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:25.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:25.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:25.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:25.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:25.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:25.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:25.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:25.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:25.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:25.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:25.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:25.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:25.278+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:25.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:25.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:25.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:25.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:26.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:26.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:26.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:26.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:27.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.150+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:14:27.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:27.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:14:27.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:27.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.162+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:27.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:27.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:27.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.186+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:27.589 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:27.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:14:27.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:27.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:27.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:27.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:27.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:27.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:27.640 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:27.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:27.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:27.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:28.138 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:28.919 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:14:28.920 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:14:28.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:28.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:28.970 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2492,"stamp":"2023-10-14T01:14:27.693977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28435307352,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2294248,"num_read_kb":8927507,"num_write":2251132,"num_write_kb":9004929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80451,"ondisk_log_size":80451,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21866692,"kb_used_data":18005932,"kb_used_omap":1331,"kb_used_meta":3859404,"kb_avail":261248828,"statfs":{"total":289910292480,"available":267518799872,"internally_reserved":0,"allocated":18438074368,"data_stored":22748129414,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1363622,"internal_metadata":3952030042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,1,1,5,10,12,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16211968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4001,"num_read_kb":15964,"num_write":3942,"num_write_kb":15768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002760"},"pg_stats":[{"pgid":"2.7","version":"145'301961","reported_seq":593205,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:26.639046+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:14:26.639046+0000","last_peered":"2023-10-14T01:14:26.639046+0000","last_clean":"2023-10-14T01:14:26.639046+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:14:26.639046+0000","last_undegraded":"2023-10-14T01:14:26.639046+0000","last_fullsized":"2023-10-14T01:14:26.639046+0000","mapping_epoch":13,"log_start":"142'291902","ondisk_log_start":"142'291902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3596624310,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":292518,"num_read_kb":1132703,"num_write":284328,"num_write_kb":1137070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'302076","reported_seq":593871,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:26.602982+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:14:26.602982+0000","last_peered":"2023-10-14T01:14:26.602982+0000","last_clean":"2023-10-14T01:14:26.602982+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:14:26.602982+0000","last_undegraded":"2023-10-14T01:14:26.602982+0000","last_fullsized":"2023-10-14T01:14:26.602982+0000","mapping_epoch":13,"log_start":"142'292011","ondisk_log_start":"142'292011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":449,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 238s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3588936048,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":294570,"num_read_kb":1130885,"num_write":284229,"num_write_kb":1136543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'273113","reported_seq":544087,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:26.613096+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:14:26.613096+0000","last_peered":"2023-10-14T01:14:26.613096+0000","last_clean":"2023-10-14T01:14:26.613096+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:14:26.613096+0000","last_undegraded":"2023-10-14T01:14:26.613096+0000","last_fullsized":"2023-10-14T01:14:26.613096+0000","mapping_epoch":13,"log_start":"142'263105","ondisk_log_start":"142'263105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3434549650,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":271399,"num_read_kb":1069816,"num_write":272495,"num_write_kb":1089950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'266821","reported_seq":534236,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:26.621995+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:14:26.621995+0000","last_peered":"2023-10-14T01:14:26.621995+0000","last_clean":"2023-10-14T01:14:26.621995+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:14:26.621995+0000","last_undegraded":"2023-10-14T01:14:26.621995+0000","last_fullsized":"2023-10-14T01:14:26.621995+0000","mapping_epoch":13,"log_start":"142'256806","ondisk_log_start":"142'256806","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":327,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 117s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3360809091,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":267823,"num_read_kb":1053725,"num_write":266033,"num_write_kb":1064054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"145'301026","reported_seq":603113,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:24.670249+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:14:24.670249+0000","last_peered":"2023-10-14T01:14:24.670249+0000","last_clean":"2023-10-14T01:14:24.670249+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:14:24.670249+0000","last_undegraded":"2023-10-14T01:14:24.670249+0000","last_fullsized":"2023-10-14T01:14:24.670249+0000","mapping_epoch":13,"log_start":"142'291009","ondisk_log_start":"142'291009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3779191103,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":302522,"num_read_kb":1184402,"num_write":298628,"num_write_kb":1194365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"145'290462","reported_seq":576623,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:27.301867+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:14:27.301867+0000","last_peered":"2023-10-14T01:14:27.301867+0000","last_clean":"2023-10-14T01:14:27.301867+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:14:27.301867+0000","last_undegraded":"2023-10-14T01:14:27.301867+0000","last_fullsized":"2023-10-14T01:14:27.301867+0000","mapping_epoch":13,"log_start":"142'280409","ondisk_log_start":"142'280409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3546538253,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":287153,"num_read_kb":1112080,"num_write":281503,"num_write_kb":1125798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"145'294874","reported_seq":588215,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:27.291294+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:14:27.291294+0000","last_peered":"2023-10-14T01:14:27.291294+0000","last_clean":"2023-10-14T01:14:27.291294+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:14:27.291294+0000","last_undegraded":"2023-10-14T01:14:27.291294+0000","last_fullsized":"2023-10-14T01:14:27.291294+0000","mapping_epoch":13,"log_start":"142'284811","ondisk_log_start":"142'284811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3595636985,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":294505,"num_read_kb":1135324,"num_write":284045,"num_write_kb":1135906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"145'289251","reported_seq":572185,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:26.622065+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:14:26.622065+0000","last_peered":"2023-10-14T01:14:26.622065+0000","last_clean":"2023-10-14T01:14:26.622065+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:14:26.622065+0000","last_undegraded":"2023-10-14T01:14:26.622065+0000","last_fullsized":"2023-10-14T01:14:26.622065+0000","mapping_epoch":13,"log_start":"142'279207","ondisk_log_start":"142'279207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":650,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 245s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3532431544,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":283632,"num_read_kb":1108463,"num_write":279702,"num_write_kb":1118695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":354,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:50.552051+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:13:50.552051+0000","last_peered":"2023-10-14T01:13:50.552051+0000","last_clean":"2023-10-14T01:13:50.552051+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:13:50.552051+0000","last_undegraded":"2023-10-14T01:13:50.552051+0000","last_fullsized":"2023-10-14T01:13:50.552051+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28434716984,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2294122,"num_read_kb":8927398,"num_write":2250963,"num_write_kb":9002381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18435694592,"data_stored":22747173488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80324,"ondisk_log_size":80324,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739307,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5251884,"kb_used_data":3381604,"kb_used_omap":214,"kb_used_meta":1870057,"kb_avail":89119956,"statfs":{"total":96636764160,"available":91258834944,"internally_reserved":0,"allocated":3462762496,"data_stored":4271904944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1914938980},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:15 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.56000000000000005,"15min":0.72199999999999998},"min":{"1min":0.17199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.58799999999999997,"5min":5.2679999999999998,"15min":7.492},"last":0.251},{"interface":"front","average":{"1min":1.0629999999999999,"5min":0.85799999999999998,"15min":0.872},"min":{"1min":0.214,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.7080000000000002,"5min":6.2080000000000002,"15min":10.847},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:14:15 2023","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.3720000000000001,"15min":1.4370000000000001},"min":{"1min":0.20799999999999999,"5min":0.193,"15min":0.191},"max":{"1min":10.726000000000001,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.653,"15min":1.3089999999999999},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":5.5839999999999996,"5min":14.586,"15min":14.586},"last":0.38300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739294,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10138212,"kb_used_data":9003164,"kb_used_omap":691,"kb_used_meta":1134348,"kb_avail":84233628,"statfs":{"total":96636764160,"available":86255235072,"internally_reserved":0,"allocated":9219239936,"data_stored":11374682442,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":1161573040},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,0,1,2,5,6,10,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.73299999999999998,"15min":0.67800000000000005},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.16800000000000001},"max":{"1min":3.1659999999999999,"5min":10.552,"15min":10.891},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.73499999999999999,"15min":0.69499999999999995},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":3.3999999999999999,"5min":10.52,"15min":10.856},"last":1.0269999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:13:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.2030000000000001,"15min":1.0600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":7.5540000000000003,"5min":8.9469999999999992,"15min":43.871000000000002},"last":7.3579999999999997},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.6279999999999999,"15min":1.2350000000000001},"min":{"1min":0.23899999999999999,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":11.06,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.44800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739305,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6476596,"kb_used_data":5621164,"kb_used_omap":425,"kb_used_meta":854998,"kb_avail":87895244,"statfs":{"total":96636764160,"available":90004729856,"internally_reserved":0,"allocated":5756071936,"data_stored":7101542028,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":875518022},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:31 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":0.92400000000000004,"15min":0.91600000000000004},"min":{"1min":0.186,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":5.258,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.57199999999999995,"15min":0.58999999999999997},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.17399999999999999},"max":{"1min":3.504,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.19900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:13:31 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.873,"15min":1.262},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":9.5809999999999995,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.006,"15min":0.79700000000000004},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":3.5,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5755494400,"data_stored":7101261434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9218568192,"data_stored":11374307640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3461632000,"data_stored":4271604414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:14:28.971 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:28.972 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:28.972 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:28.973 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:28.973 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:28.973 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:29.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:29.030+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:31.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:14:31.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:14:31.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:14:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:14:31.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:31.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:31.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:31.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:31.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:31.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:32.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:32.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:32.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:32.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:32.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:32.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:32.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:32.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:33.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:33.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:33.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:33.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:33.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:33.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:33.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:33.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:33.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:33.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:33.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:33.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:34.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:34.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:34.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:34.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:34.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:34.546+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:34.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:34.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:34.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:34.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:34.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:34.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:34.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:34.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:35.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:35.662+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:35.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:35.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:35.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:35.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:35.723 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:35.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:35.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:35.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:35.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:35.718+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:37.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:37.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:37.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:37.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:37.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:37.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:37.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:37.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:40.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:40.286+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:40.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:40.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:40.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:40.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:41.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:41.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:41.076 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:41.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:41.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:41.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:41.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:41.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:41.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:41.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:41.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:41.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:41.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:41.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:41.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:41.542+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:42.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:42.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:42.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:42.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:42.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:42.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:42.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:42.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:42.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:42.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:42.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:42.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:44.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:44.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:14:44.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:44.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:44.846 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:44.842+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:44.865 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:44.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:44.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:44.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:44.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:44.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:44.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:44.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:45.972 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:45.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:45.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:45.974+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:45.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:45.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:45.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:45.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:46.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:46.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:46.193 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:46.190+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:46.199 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:46.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:46.200 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:46.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:46.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:46.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:46.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:46.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:46.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:46.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:46.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.282+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:46.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:46.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:46.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:46.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:46.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:46.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:47.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:14:47.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:14:47.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:14:47.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:47.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:14:47.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:14:47.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:47.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:47.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:47.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:47.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:14:47.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:14:47.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:14:47.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:47.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:47.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:47.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:48.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:48.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:48.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:48.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:48.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:48.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:48.974 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:49.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:49.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:49.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:49.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:49.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:49.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:49.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:49.354+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:50.037 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:14:50.038 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:14:50.110 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2503,"stamp":"2023-10-14T01:14:49.701944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28468104024,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2302159,"num_read_kb":8959063,"num_write":2259161,"num_write_kb":9037045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80488,"ondisk_log_size":80488,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22142776,"kb_used_data":18071192,"kb_used_omap":1331,"kb_used_meta":4070220,"kb_avail":260972744,"statfs":{"total":289910292480,"available":267236089856,"internally_reserved":0,"allocated":18504900608,"data_stored":22814955654,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1363622,"internal_metadata":4167905626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,7,7,17,22,34,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16318464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3917,"num_read_kb":15624,"num_write":3993,"num_write_kb":15972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006308"},"pg_stats":[{"pgid":"2.7","version":"145'303084","reported_seq":595318,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:47.356503+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:14:47.356503+0000","last_peered":"2023-10-14T01:14:47.356503+0000","last_clean":"2023-10-14T01:14:47.356503+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:14:47.356503+0000","last_undegraded":"2023-10-14T01:14:47.356503+0000","last_fullsized":"2023-10-14T01:14:47.356503+0000","mapping_epoch":13,"log_start":"142'293002","ondisk_log_start":"142'293002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3600892342,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":293492,"num_read_kb":1136583,"num_write":285382,"num_write_kb":1141286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'303132","reported_seq":595993,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:47.300838+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:14:47.300838+0000","last_peered":"2023-10-14T01:14:47.300838+0000","last_clean":"2023-10-14T01:14:47.300838+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:14:47.300838+0000","last_undegraded":"2023-10-14T01:14:47.300838+0000","last_fullsized":"2023-10-14T01:14:47.300838+0000","mapping_epoch":13,"log_start":"142'293112","ondisk_log_start":"142'293112","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":504,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 259s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3592925552,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":295611,"num_read_kb":1135045,"num_write":285195,"num_write_kb":1140407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'274033","reported_seq":546014,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:47.335798+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:14:47.335798+0000","last_peered":"2023-10-14T01:14:47.335798+0000","last_clean":"2023-10-14T01:14:47.335798+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:14:47.335798+0000","last_undegraded":"2023-10-14T01:14:47.335798+0000","last_fullsized":"2023-10-14T01:14:47.335798+0000","mapping_epoch":13,"log_start":"142'264005","ondisk_log_start":"142'264005","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3438313874,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":272405,"num_read_kb":1073832,"num_write":273415,"num_write_kb":1093630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'267779","reported_seq":536229,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:47.297250+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:14:47.297250+0000","last_peered":"2023-10-14T01:14:47.297250+0000","last_clean":"2023-10-14T01:14:47.297250+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:14:47.297250+0000","last_undegraded":"2023-10-14T01:14:47.297250+0000","last_fullsized":"2023-10-14T01:14:47.297250+0000","mapping_epoch":13,"log_start":"142'257706","ondisk_log_start":"142'257706","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":400,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 138s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3364561027,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":268849,"num_read_kb":1057821,"num_write":266953,"num_write_kb":1067734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"145'302141","reported_seq":605215,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:44.897518+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:14:44.897518+0000","last_peered":"2023-10-14T01:14:44.897518+0000","last_clean":"2023-10-14T01:14:44.897518+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:14:44.897518+0000","last_undegraded":"2023-10-14T01:14:44.897518+0000","last_fullsized":"2023-10-14T01:14:44.897518+0000","mapping_epoch":13,"log_start":"142'292109","ondisk_log_start":"142'292109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3783737663,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":303501,"num_read_kb":1188306,"num_write":299740,"num_write_kb":1198813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"145'291518","reported_seq":578611,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:47.407653+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:14:47.407653+0000","last_peered":"2023-10-14T01:14:47.407653+0000","last_clean":"2023-10-14T01:14:47.407653+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:14:47.407653+0000","last_undegraded":"2023-10-14T01:14:47.407653+0000","last_fullsized":"2023-10-14T01:14:47.407653+0000","mapping_epoch":13,"log_start":"142'281509","ondisk_log_start":"142'281509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3550769421,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":288074,"num_read_kb":1115740,"num_write":282535,"num_write_kb":1129926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"145'295901","reported_seq":590236,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:47.397502+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:14:47.397502+0000","last_peered":"2023-10-14T01:14:47.397502+0000","last_clean":"2023-10-14T01:14:47.397502+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:14:47.397502+0000","last_undegraded":"2023-10-14T01:14:47.397502+0000","last_fullsized":"2023-10-14T01:14:47.397502+0000","mapping_epoch":13,"log_start":"142'285811","ondisk_log_start":"142'285811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3599700217,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":295484,"num_read_kb":1139232,"num_write":285042,"num_write_kb":1139894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"145'290335","reported_seq":574270,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:47.336402+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:14:47.336402+0000","last_peered":"2023-10-14T01:14:47.336402+0000","last_clean":"2023-10-14T01:14:47.336402+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:14:47.336402+0000","last_undegraded":"2023-10-14T01:14:47.336402+0000","last_fullsized":"2023-10-14T01:14:47.336402+0000","mapping_epoch":13,"log_start":"142'280308","ondisk_log_start":"142'280308","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":690,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 266s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3536613560,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":284617,"num_read_kb":1112395,"num_write":280730,"num_write_kb":1122807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":354,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:50.552051+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:13:50.552051+0000","last_peered":"2023-10-14T01:13:50.552051+0000","last_clean":"2023-10-14T01:13:50.552051+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:13:50.552051+0000","last_undegraded":"2023-10-14T01:13:50.552051+0000","last_fullsized":"2023-10-14T01:13:50.552051+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28467513656,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2302033,"num_read_kb":8958954,"num_write":2258992,"num_write_kb":9034497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18501423104,"data_stored":22812902000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80361,"ondisk_log_size":80361,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739311,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5312388,"kb_used_data":3393528,"kb_used_omap":214,"kb_used_meta":1918633,"kb_avail":89059452,"statfs":{"total":96636764160,"available":91196878848,"internally_reserved":0,"allocated":3474972672,"data_stored":4284115120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":1964680804},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,3,7,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:15 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.56000000000000005,"15min":0.72199999999999998},"min":{"1min":0.17199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.58799999999999997,"5min":5.2679999999999998,"15min":7.492},"last":4.7050000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":0.85799999999999998,"15min":0.872},"min":{"1min":0.214,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.7080000000000002,"5min":6.2080000000000002,"15min":10.847},"last":0.37}]},{"osd":1,"last update":"Sat Oct 14 01:14:15 2023","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.3720000000000001,"15min":1.4370000000000001},"min":{"1min":0.20799999999999999,"5min":0.193,"15min":0.191},"max":{"1min":10.726000000000001,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.247},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.653,"15min":1.3089999999999999},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":5.5839999999999996,"5min":14.586,"15min":14.586},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739298,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10257256,"kb_used_data":9036256,"kb_used_omap":691,"kb_used_meta":1220300,"kb_avail":84114584,"statfs":{"total":96636764160,"available":86133334016,"internally_reserved":0,"allocated":9253126144,"data_stored":11408568650,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":1249587888},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,4,10,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65600000000000003,"15min":0.67500000000000004},"min":{"1min":0.23200000000000001,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":3.9940000000000002,"5min":10.552,"15min":10.891},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.625,"15min":0.68100000000000005},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.1319999999999999,"5min":10.52,"15min":10.856},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:13:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":1.2030000000000001,"15min":1.0600000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":7.5540000000000003,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.9570000000000001,"5min":1.6279999999999999,"15min":1.2350000000000001},"min":{"1min":0.23899999999999999,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":11.06,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.23000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739309,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6573132,"kb_used_data":5641408,"kb_used_omap":425,"kb_used_meta":931286,"kb_avail":87798708,"statfs":{"total":96636764160,"available":89905876992,"internally_reserved":0,"allocated":5776801792,"data_stored":7122271884,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":953636934},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,4,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.91400000000000003,"15min":0.94499999999999995},"min":{"1min":0.251,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.919,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":5.351,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.56000000000000005}]},{"osd":2,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.77200000000000002,"15min":1.175},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":2.2490000000000001,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.97699999999999998,"15min":0.78400000000000003},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":1.599,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5775507456,"data_stored":7121274490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9251885056,"data_stored":11407624504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3474030592,"data_stored":4284003006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:14:50.112 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:50.112 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:50.112 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:50.113 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:50.113 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:50.114 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:14:50.834 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:14:50.906 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-14T01:14:51.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:51.462+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:51.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:51.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:51.744 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-14T01:14:51.779 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:14:51.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:51.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:51.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:51.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:51.946+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:52.507 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:14:52.552 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-14T01:14:53.447 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-14T01:14:53.484 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-14T01:14:54.122 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-14T01:14:54.197 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:54.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:54.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:54.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:54.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:54.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:54.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:54.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:54.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:54.957 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:54.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:54.958 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:54.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:55.281 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:55.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:55.330 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:55.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:55.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:55.342+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:55.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:55.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:55.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:55.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:55.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:55.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:55.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:55.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:56.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:56.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:56.664 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:56.658+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:57.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:57.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:57.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:57.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:14:57.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:57.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:58.977 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:58.974+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:58.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:58.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:59.005 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:59.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:59.006 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.002+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:59.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:59.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:14:59.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:59.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:14:59.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:14:59.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:14:59.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:14:59.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:14:59.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:14:59.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:14:59.230+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:14:59.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:59.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:14:59.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:14:59.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:00.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:00.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:00.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:00.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:00.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:00.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:15:00.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:00.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:00.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:00.978+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:01.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:00.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:01.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:00.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:01.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:01.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:01.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:01.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:01.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:01.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:01.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:01.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:01.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:01.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:01.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:01.514+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:01.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:01.526+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:01.539 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:01.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:01.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:01.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:01.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:01.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:01.591 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:01.586+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:01.881 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:15:01.878+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:15:02.717 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:02.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:02.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:02.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:02.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:02.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:02.738+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:03.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:03.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:03.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:03.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:04.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:04.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:04.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:04.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:04.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:04.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:04.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:04.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:04.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:04.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:04.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:04.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:05.537 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:15:05.537 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.534+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:05.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:05.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:05.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:05.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:05.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:05.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.666+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:05.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:05.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:05.721 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:05.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:05.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:05.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:05.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:05.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:05.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:05.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:06.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:15:06.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:15:06.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:06.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:06.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:06.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:06.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:06.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:06.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:06.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:06.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:06.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:06.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.854+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:06.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:06.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:15:06.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:15:06.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:15:06.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:06.920 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:06.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:06.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:06.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:06.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:07.152 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:07.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:07.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:07.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:07.198 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:07.194+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:07.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:07.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:07.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:07.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:07.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:07.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:07.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:07.453 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:07.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:07.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:07.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.482+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:07.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:07.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:07.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:07.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:07.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:08.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:08.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:08.830 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:08.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:08.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:08.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:08.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:08.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:09.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:09.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:09.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:09.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:09.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:09.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:10.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:10.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:10.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:10.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:10.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:10.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:10.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:10.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:10.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:10.442+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:11.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:11.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:11.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:11.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:11.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:11.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:12.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:12.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:15:12.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:12.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:12.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:12.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:12.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:12.277+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:12.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:12.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:12.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:12.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:12.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:12.289+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:12.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:12.709+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:12.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:12.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:12.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:12.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:13.323 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:13.317+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:13.340 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:13.333+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:13.373 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:13.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:13.373 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:13.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:13.374 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:13.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:14.198 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:14.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:14.265+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:14.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:14.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:14.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:14.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:15.179 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:15:15.180 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:15:15.220 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2515,"stamp":"2023-10-14T01:15:13.705795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28508375896,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2311828,"num_read_kb":8997651,"num_write":2268963,"num_write_kb":9076253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80455,"ondisk_log_size":80455,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22278076,"kb_used_data":18148384,"kb_used_omap":1331,"kb_used_meta":4128332,"kb_avail":260837444,"statfs":{"total":289910292480,"available":267097542656,"internally_reserved":0,"allocated":18583945216,"data_stored":22894000262,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1363622,"internal_metadata":4227412314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,3,4,9,13,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23113728,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5506,"num_read_kb":21972,"num_write":5621,"num_write_kb":22484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001882"},"pg_stats":[{"pgid":"2.7","version":"145'304476","reported_seq":597944,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:15:13.365383+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:15:13.365383+0000","last_peered":"2023-10-14T01:15:13.365383+0000","last_clean":"2023-10-14T01:15:13.365383+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:15:13.365383+0000","last_undegraded":"2023-10-14T01:15:13.365383+0000","last_fullsized":"2023-10-14T01:15:13.365383+0000","mapping_epoch":13,"log_start":"143'294402","ondisk_log_start":"143'294402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3606184374,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":294703,"num_read_kb":1141407,"num_write":286673,"num_write_kb":1146450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"145'304545","reported_seq":598579,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:13.334424+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:15:13.334424+0000","last_peered":"2023-10-14T01:15:13.334424+0000","last_clean":"2023-10-14T01:15:13.334424+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:15:13.334424+0000","last_undegraded":"2023-10-14T01:15:13.334424+0000","last_fullsized":"2023-10-14T01:15:13.334424+0000","mapping_epoch":13,"log_start":"143'294512","ondisk_log_start":"143'294512","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":550,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 285s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3598193008,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":296757,"num_read_kb":1139617,"num_write":286483,"num_write_kb":1145559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'275217","reported_seq":548403,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:15:13.324130+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:15:13.324130+0000","last_peered":"2023-10-14T01:15:13.324130+0000","last_clean":"2023-10-14T01:15:13.324130+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:15:13.324130+0000","last_undegraded":"2023-10-14T01:15:13.324130+0000","last_fullsized":"2023-10-14T01:15:13.324130+0000","mapping_epoch":13,"log_start":"142'265205","ondisk_log_start":"142'265205","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3443179922,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":273608,"num_read_kb":1078636,"num_write":274599,"num_write_kb":1098366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"145'268941","reported_seq":538590,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:13.291341+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:15:13.291341+0000","last_peered":"2023-10-14T01:15:13.291341+0000","last_clean":"2023-10-14T01:15:13.291341+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:15:13.291341+0000","last_undegraded":"2023-10-14T01:15:13.291341+0000","last_fullsized":"2023-10-14T01:15:13.291341+0000","mapping_epoch":13,"log_start":"142'258907","ondisk_log_start":"142'258907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":470,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 164s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3369189507,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":270035,"num_read_kb":1062553,"num_write":268078,"num_write_kb":1072234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"145'303414","reported_seq":607774,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:15:10.174131+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:15:10.174131+0000","last_peered":"2023-10-14T01:15:10.174131+0000","last_clean":"2023-10-14T01:15:10.174131+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:15:10.174131+0000","last_undegraded":"2023-10-14T01:15:10.174131+0000","last_fullsized":"2023-10-14T01:15:10.174131+0000","mapping_epoch":13,"log_start":"143'293409","ondisk_log_start":"143'293409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3788931391,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":304786,"num_read_kb":1193434,"num_write":301009,"num_write_kb":1203889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"145'292794","reported_seq":581062,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:15:12.670707+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:15:12.670707+0000","last_peered":"2023-10-14T01:15:12.670707+0000","last_clean":"2023-10-14T01:15:12.670707+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:15:12.670707+0000","last_undegraded":"2023-10-14T01:15:12.670707+0000","last_fullsized":"2023-10-14T01:15:12.670707+0000","mapping_epoch":13,"log_start":"142'282709","ondisk_log_start":"142'282709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3555701005,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":289229,"num_read_kb":1120360,"num_write":283755,"num_write_kb":1134806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"145'297167","reported_seq":592773,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:15:12.676766+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:15:12.676766+0000","last_peered":"2023-10-14T01:15:12.676766+0000","last_clean":"2023-10-14T01:15:12.676766+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:15:12.676766+0000","last_undegraded":"2023-10-14T01:15:12.676766+0000","last_fullsized":"2023-10-14T01:15:12.676766+0000","mapping_epoch":13,"log_start":"143'287111","ondisk_log_start":"143'287111","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3604746489,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":296734,"num_read_kb":1144220,"num_write":286238,"num_write_kb":1144678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"145'291638","reported_seq":576829,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:13.325608+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:15:13.325608+0000","last_peered":"2023-10-14T01:15:13.325608+0000","last_clean":"2023-10-14T01:15:13.325608+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:15:13.325608+0000","last_undegraded":"2023-10-14T01:15:13.325608+0000","last_fullsized":"2023-10-14T01:15:13.325608+0000","mapping_epoch":13,"log_start":"143'281609","ondisk_log_start":"143'281609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":740,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 292s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3541659832,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":285850,"num_read_kb":1117315,"num_write":281959,"num_write_kb":1127723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":354,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-14T01:13:50.552051+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:13:50.552051+0000","last_peered":"2023-10-14T01:13:50.552051+0000","last_clean":"2023-10-14T01:13:50.552051+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:13:50.552051+0000","last_undegraded":"2023-10-14T01:13:50.552051+0000","last_fullsized":"2023-10-14T01:13:50.552051+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28507785528,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2311702,"num_read_kb":8997542,"num_write":2268794,"num_write_kb":9073705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18581143552,"data_stored":22892622448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80328,"ondisk_log_size":80328,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739316,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5376744,"kb_used_data":3407904,"kb_used_omap":214,"kb_used_meta":1968617,"kb_avail":88995096,"statfs":{"total":96636764160,"available":91130978304,"internally_reserved":0,"allocated":3489693696,"data_stored":4298836144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219548,"internal_metadata":2015864420},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,3,15,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:15 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.56000000000000005,"15min":0.72199999999999998},"min":{"1min":0.17199999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.58799999999999997,"5min":5.2679999999999998,"15min":7.492},"last":0.25},{"interface":"front","average":{"1min":1.0629999999999999,"5min":0.85799999999999998,"15min":0.872},"min":{"1min":0.214,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.7080000000000002,"5min":6.2080000000000002,"15min":10.847},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:14:15 2023","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.3720000000000001,"15min":1.4370000000000001},"min":{"1min":0.20799999999999999,"5min":0.193,"15min":0.191},"max":{"1min":10.726000000000001,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.379},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.653,"15min":1.3089999999999999},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":5.5839999999999996,"5min":14.586,"15min":14.586},"last":4.8760000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739303,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10448724,"kb_used_data":9075912,"kb_used_omap":691,"kb_used_meta":1372108,"kb_avail":83923116,"statfs":{"total":96636764160,"available":85937270784,"internally_reserved":0,"allocated":9293733888,"data_stored":11449176394,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":1405039280},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,1,3,8,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65600000000000003,"15min":0.67500000000000004},"min":{"1min":0.23200000000000001,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":3.9940000000000002,"5min":10.552,"15min":10.891},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.625,"15min":0.68100000000000005},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.1319999999999999,"5min":10.52,"15min":10.856},"last":0.245}]},{"osd":2,"last update":"Sat Oct 14 01:14:49 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.083,"15min":1.075},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":7.3579999999999997,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.505,"5min":1.4410000000000001,"15min":1.228},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.2530000000000001,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.22800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739314,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6452608,"kb_used_data":5664568,"kb_used_omap":425,"kb_used_meta":787606,"kb_avail":87919232,"statfs":{"total":96636764160,"available":90029293568,"internally_reserved":0,"allocated":5800517632,"data_stored":7145987724,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":806508614},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.91400000000000003,"15min":0.94499999999999995},"min":{"1min":0.251,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.919,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":5.351,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.20699999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.77200000000000002,"15min":1.175},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":2.2490000000000001,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.97699999999999998,"15min":0.78400000000000003},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":1.599,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5799911424,"data_stored":7145678458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9292365824,"data_stored":11448105272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3488866304,"data_stored":4298838718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:15:15.221 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:15.222 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:15.222 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:15.222 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:15.223 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:15.223 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:15.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:15.517+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:15.533 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:15.529+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:15.579 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:15.573+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:15.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:15.589+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:15.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:15.589+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:15.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:15.589+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:15.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:15.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:15.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:15.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:15.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:15.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:15.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:15.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:17.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:17.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:17.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:17.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:19.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:19.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:19.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:19.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:19.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:19.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:19.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:19.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:19.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:19.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:19.722 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:19.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:15:19.723 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:19.717+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:19.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:19.725+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:19.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:19.725+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:19.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:19.725+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:19.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:19.725+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:19.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:19.725+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:20.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:15:20.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:20.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:20.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:20.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:20.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:20.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:20.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:20.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:23.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:23.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:23.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:23.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:23.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:23.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:23.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:23.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:23.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:23.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:23.564 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:23.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:23.607 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:23.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:23.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:23.617+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:23.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:23.617+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:24.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:24.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:24.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:24.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:24.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:24.565+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:24.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:24.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:24.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:24.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:27.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:27.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:27.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:27.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:27.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:27.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:27.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:27.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:27.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:27.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:28.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:28.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:28.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:28.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:28.835 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:28.829+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:15:28.837 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:28.833+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:28.845 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:28.841+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:15:28.885 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:28.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:15:28.886 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:28.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:15:28.886 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:28.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:15:28.887 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:28.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:15:29.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:29.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:29.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:29.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:29.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:29.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:31.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:31.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:31.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:31.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:31.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:31.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:31.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:31.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:31.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:31.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:31.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:31.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:31.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:31.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:32.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:32.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:32.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:32.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:32.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:32.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:33.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:33.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:33.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:33.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:33.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:33.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:34.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:34.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:34.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:34.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:34.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:34.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:34.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:34.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:34.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:34.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:34.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:34.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:35.225 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:36.183 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:15:36.184 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:15:36.242 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2527,"stamp":"2023-10-14T01:15:35.715766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28542077784,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2319965,"num_read_kb":9030115,"num_write":2277180,"num_write_kb":9109121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80587,"ondisk_log_size":80587,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21392692,"kb_used_data":18216280,"kb_used_omap":1434,"kb_used_meta":3174949,"kb_avail":261722828,"statfs":{"total":289910292480,"available":268004175872,"internally_reserved":0,"allocated":18653470720,"data_stored":22963514198,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1469044,"internal_metadata":3251148172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,9,9,8,12,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15917056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3920,"num_read_kb":15644,"num_write":3886,"num_write_kb":15544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001642"},"pg_stats":[{"pgid":"2.7","version":"146'305582","reported_seq":600095,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:34.199636+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:15:34.199636+0000","last_peered":"2023-10-14T01:15:34.199636+0000","last_clean":"2023-10-14T01:15:34.199636+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:15:34.199636+0000","last_undegraded":"2023-10-14T01:15:34.199636+0000","last_fullsized":"2023-10-14T01:15:34.199636+0000","mapping_epoch":13,"log_start":"143'295502","ondisk_log_start":"143'295502","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3610423734,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":295729,"num_read_kb":1145503,"num_write":287711,"num_write_kb":1150602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'305628","reported_seq":600653,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:34.202751+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:15:34.202751+0000","last_peered":"2023-10-14T01:15:34.202751+0000","last_clean":"2023-10-14T01:15:34.202751+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:15:34.202751+0000","last_undegraded":"2023-10-14T01:15:34.202751+0000","last_fullsized":"2023-10-14T01:15:34.202751+0000","mapping_epoch":13,"log_start":"143'295612","ondisk_log_start":"143'295612","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":606,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 306s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3602219376,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":297718,"num_read_kb":1143457,"num_write":287470,"num_write_kb":1149507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'276192","reported_seq":550316,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:34.158541+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:15:34.158541+0000","last_peered":"2023-10-14T01:15:34.158541+0000","last_clean":"2023-10-14T01:15:34.158541+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:15:34.158541+0000","last_undegraded":"2023-10-14T01:15:34.158541+0000","last_fullsized":"2023-10-14T01:15:34.158541+0000","mapping_epoch":13,"log_start":"143'266105","ondisk_log_start":"143'266105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3447202194,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":274546,"num_read_kb":1082376,"num_write":275574,"num_write_kb":1102266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'269943","reported_seq":540521,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:34.122663+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:15:34.122663+0000","last_peered":"2023-10-14T01:15:34.122663+0000","last_clean":"2023-10-14T01:15:34.122663+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:15:34.122663+0000","last_undegraded":"2023-10-14T01:15:34.122663+0000","last_fullsized":"2023-10-14T01:15:34.122663+0000","mapping_epoch":13,"log_start":"143'259907","ondisk_log_start":"143'259907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":526,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 184s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3373162627,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":270952,"num_read_kb":1066205,"num_write":269051,"num_write_kb":1076126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"146'304699","reported_seq":610379,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:35.540284+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:15:35.540284+0000","last_peered":"2023-10-14T01:15:35.540284+0000","last_clean":"2023-10-14T01:15:35.540284+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:15:35.540284+0000","last_undegraded":"2023-10-14T01:15:35.540284+0000","last_fullsized":"2023-10-14T01:15:35.540284+0000","mapping_epoch":13,"log_start":"143'294609","ondisk_log_start":"143'294609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3794194751,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":306100,"num_read_kb":1198678,"num_write":302292,"num_write_kb":1209021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"146'293823","reported_seq":583054,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:32.736031+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:15:32.736031+0000","last_peered":"2023-10-14T01:15:32.736031+0000","last_clean":"2023-10-14T01:15:32.736031+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:15:32.736031+0000","last_undegraded":"2023-10-14T01:15:32.736031+0000","last_fullsized":"2023-10-14T01:15:32.736031+0000","mapping_epoch":13,"log_start":"143'283809","ondisk_log_start":"143'283809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3559846157,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":290174,"num_read_kb":1124136,"num_write":284756,"num_write_kb":1138810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"146'298199","reported_seq":594819,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:32.768324+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:15:32.768324+0000","last_peered":"2023-10-14T01:15:32.768324+0000","last_clean":"2023-10-14T01:15:32.768324+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:15:32.768324+0000","last_undegraded":"2023-10-14T01:15:32.768324+0000","last_fullsized":"2023-10-14T01:15:32.768324+0000","mapping_epoch":13,"log_start":"143'288111","ondisk_log_start":"143'288111","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3608805625,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":297724,"num_read_kb":1148168,"num_write":287235,"num_write_kb":1148666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"146'292658","reported_seq":578913,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:34.125431+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:15:34.125431+0000","last_peered":"2023-10-14T01:15:34.125431+0000","last_clean":"2023-10-14T01:15:34.125431+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:15:34.125431+0000","last_undegraded":"2023-10-14T01:15:34.125431+0000","last_fullsized":"2023-10-14T01:15:34.125431+0000","mapping_epoch":13,"log_start":"143'282609","ondisk_log_start":"143'282609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":782,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 313s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3545632952,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":286896,"num_read_kb":1121483,"num_write":282922,"num_write_kb":1131575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":355,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-10-14T01:14:19.641720+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:14:19.641720+0000","last_peered":"2023-10-14T01:14:19.641720+0000","last_clean":"2023-10-14T01:14:19.641720+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:14:19.641720+0000","last_undegraded":"2023-10-14T01:14:19.641720+0000","last_fullsized":"2023-10-14T01:14:19.641720+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28541487416,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2319839,"num_read_kb":9030006,"num_write":2277011,"num_write_kb":9106573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18651402240,"data_stored":22962881136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80460,"ondisk_log_size":80460,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4193192,"kb_used_data":3419548,"kb_used_omap":317,"kb_used_meta":773314,"kb_avail":90178648,"statfs":{"total":96636764160,"available":92342935552,"internally_reserved":0,"allocated":3501617152,"data_stored":4310755744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":791874198},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:18 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.68700000000000006,"15min":0.73799999999999999},"min":{"1min":0.20899999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":7.492},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.81299999999999994,"15min":0.85299999999999998},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.7250000000000001,"5min":6.2080000000000002,"15min":10.847},"last":0.23499999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:15:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.403,"15min":1.377},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.191},"max":{"1min":6.0919999999999996,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.601,"15min":1.2849999999999999},"min":{"1min":0.22800000000000001,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":4.8760000000000003,"5min":14.586,"15min":14.586},"last":2.9180000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739307,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10639840,"kb_used_data":9106968,"kb_used_omap":691,"kb_used_meta":1532172,"kb_avail":83732000,"statfs":{"total":96636764160,"available":85741568000,"internally_reserved":0,"allocated":9325535232,"data_stored":11480973882,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":1568944816},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,4,4,8,4,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65600000000000003,"15min":0.67500000000000004},"min":{"1min":0.23200000000000001,"5min":0.20100000000000001,"15min":0.186},"max":{"1min":3.9940000000000002,"5min":10.552,"15min":10.891},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.625,"15min":0.68100000000000005},"min":{"1min":0.26700000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.1319999999999999,"5min":10.52,"15min":10.856},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:14:49 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.083,"15min":1.075},"min":{"1min":0.251,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":7.3579999999999997,"5min":8.9469999999999992,"15min":43.871000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.505,"5min":1.4410000000000001,"15min":1.228},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":2.2530000000000001,"5min":11.412000000000001,"15min":39.412999999999997},"last":0.48499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739319,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6559660,"kb_used_data":5689764,"kb_used_omap":425,"kb_used_meta":869462,"kb_avail":87812180,"statfs":{"total":96636764160,"available":89919672320,"internally_reserved":0,"allocated":5826318336,"data_stored":7171784572,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":890329158},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.91400000000000003,"15min":0.94499999999999995},"min":{"1min":0.251,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":3.919,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.25},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":5.351,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.26500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:14:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.77200000000000002,"15min":1.175},"min":{"1min":0.215,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":2.2490000000000001,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.97699999999999998,"15min":0.78400000000000003},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":1.599,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5825077248,"data_stored":7170844282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9325502464,"data_stored":11481241912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3500822528,"data_stored":4310794942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:15:36.244 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:36.244 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:36.245 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:36.245 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:36.246 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:36.246 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:37.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:37.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:37.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:37.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:37.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:37.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:37.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:37.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:37.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:37.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:37.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:37.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:37.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:37.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:38.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:38.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:38.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:38.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:38.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:38.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:38.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:15:38.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:15:38.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:38.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:38.890 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:38.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:38.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:38.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:39.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:15:39.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:15:39.393 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:15:39.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:39.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:39.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:39.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:39.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:39.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:39.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:39.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:15:39.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:15:39.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:39.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:39.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:39.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:39.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:43.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:43.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:43.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:43.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:43.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:43.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:43.491 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:43.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:43.738 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:15:43.737+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:15:44.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:44.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:44.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:44.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:44.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:44.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:45.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:45.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:45.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:15:45.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:45.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:45.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:45.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:45.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:45.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:45.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:47.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:47.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:47.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:47.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:48.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:48.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:48.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:48.929+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:48.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:48.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:49.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:49.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:49.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:49.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:15:49.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:15:49.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:15:49.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.905+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:49.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:49.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:49.919 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:49.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:49.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:49.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:49.933+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:50.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:50.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:50.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:50.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:50.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:50.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:50.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:50.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:50.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:50.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:53.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:53.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:53.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:53.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:53.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:53.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:53.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:53.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:55.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:55.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:15:55.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:55.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:15:55.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:55.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:15:55.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:55.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:55.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:55.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:56.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:56.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:56.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:56.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:56.247 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:57.295 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:15:57.296 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:15:57.387 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2537,"stamp":"2023-10-14T01:15:55.719361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28573580120,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2327615,"num_read_kb":9060635,"num_write":2284898,"num_write_kb":9139993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80585,"ondisk_log_size":80585,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21669552,"kb_used_data":18276052,"kb_used_omap":1434,"kb_used_meta":3392037,"kb_avail":261445968,"statfs":{"total":289910292480,"available":267720671232,"internally_reserved":0,"allocated":18714677248,"data_stored":23024720726,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1469044,"internal_metadata":3473446284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,3,5,15,15,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21299200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5182,"num_read_kb":20676,"num_write":5221,"num_write_kb":20884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002009"},"pg_stats":[{"pgid":"2.7","version":"146'306693","reported_seq":602214,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:54.982499+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:15:54.982499+0000","last_peered":"2023-10-14T01:15:54.982499+0000","last_clean":"2023-10-14T01:15:54.982499+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:15:54.982499+0000","last_undegraded":"2023-10-14T01:15:54.982499+0000","last_fullsized":"2023-10-14T01:15:54.982499+0000","mapping_epoch":13,"log_start":"143'296602","ondisk_log_start":"143'296602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3614699958,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":296718,"num_read_kb":1149451,"num_write":288754,"num_write_kb":1154774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'306703","reported_seq":602756,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:55.003725+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:15:55.003725+0000","last_peered":"2023-10-14T01:15:55.003725+0000","last_clean":"2023-10-14T01:15:55.003725+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:15:55.003725+0000","last_undegraded":"2023-10-14T01:15:55.003725+0000","last_fullsized":"2023-10-14T01:15:55.003725+0000","mapping_epoch":13,"log_start":"143'296612","ondisk_log_start":"143'296612","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":634,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 326s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3606266224,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":298721,"num_read_kb":1147465,"num_write":288461,"num_write_kb":1153471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'277171","reported_seq":552225,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:54.973215+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:15:54.973215+0000","last_peered":"2023-10-14T01:15:54.973215+0000","last_clean":"2023-10-14T01:15:54.973215+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:15:54.973215+0000","last_undegraded":"2023-10-14T01:15:54.973215+0000","last_fullsized":"2023-10-14T01:15:54.973215+0000","mapping_epoch":13,"log_start":"143'267105","ondisk_log_start":"143'267105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3451179410,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":275480,"num_read_kb":1086100,"num_write":276553,"num_write_kb":1106182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'270912","reported_seq":542444,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:54.964269+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:15:54.964269+0000","last_peered":"2023-10-14T01:15:54.964269+0000","last_clean":"2023-10-14T01:15:54.964269+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:15:54.964269+0000","last_undegraded":"2023-10-14T01:15:54.964269+0000","last_fullsized":"2023-10-14T01:15:54.964269+0000","mapping_epoch":13,"log_start":"143'260907","ondisk_log_start":"143'260907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":596,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 205s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3376959619,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":271902,"num_read_kb":1069989,"num_write":269981,"num_write_kb":1079846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"146'305489","reported_seq":611956,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:50.707883+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:15:50.707883+0000","last_peered":"2023-10-14T01:15:50.707883+0000","last_clean":"2023-10-14T01:15:50.707883+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:15:50.707883+0000","last_undegraded":"2023-10-14T01:15:50.707883+0000","last_fullsized":"2023-10-14T01:15:50.707883+0000","mapping_epoch":13,"log_start":"143'295409","ondisk_log_start":"143'295409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3797397823,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":306882,"num_read_kb":1201802,"num_write":303080,"num_write_kb":1212173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"146'294815","reported_seq":585040,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:53.075128+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:15:53.075128+0000","last_peered":"2023-10-14T01:15:53.075128+0000","last_clean":"2023-10-14T01:15:53.075128+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:15:53.075128+0000","last_undegraded":"2023-10-14T01:15:53.075128+0000","last_fullsized":"2023-10-14T01:15:53.075128+0000","mapping_epoch":13,"log_start":"143'284809","ondisk_log_start":"143'284809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3563827469,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":291155,"num_read_kb":1128040,"num_write":285735,"num_write_kb":1142726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"146'299271","reported_seq":596934,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:53.095495+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:15:53.095495+0000","last_peered":"2023-10-14T01:15:53.095495+0000","last_clean":"2023-10-14T01:15:53.095495+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:15:53.095495+0000","last_undegraded":"2023-10-14T01:15:53.095495+0000","last_fullsized":"2023-10-14T01:15:53.095495+0000","mapping_epoch":13,"log_start":"143'289211","ondisk_log_start":"143'289211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3613147385,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":298746,"num_read_kb":1152252,"num_write":288292,"num_write_kb":1152894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"146'293668","reported_seq":580932,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:54.922863+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:15:54.922863+0000","last_peered":"2023-10-14T01:15:54.922863+0000","last_clean":"2023-10-14T01:15:54.922863+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:15:54.922863+0000","last_undegraded":"2023-10-14T01:15:54.922863+0000","last_fullsized":"2023-10-14T01:15:54.922863+0000","mapping_epoch":13,"log_start":"143'283609","ondisk_log_start":"143'283609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":828,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 333s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3549511864,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":287885,"num_read_kb":1125427,"num_write":283873,"num_write_kb":1135379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":356,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.689327+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:15:29.689327+0000","last_peered":"2023-10-14T01:15:29.689327+0000","last_clean":"2023-10-14T01:15:29.689327+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:15:29.689327+0000","last_undegraded":"2023-10-14T01:15:29.689327+0000","last_fullsized":"2023-10-14T01:15:29.689327+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28572989752,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2327489,"num_read_kb":9060526,"num_write":2284729,"num_write_kb":9137445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18711662592,"data_stored":23023141488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80458,"ondisk_log_size":80458,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739324,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4237476,"kb_used_data":3431900,"kb_used_omap":317,"kb_used_meta":805250,"kb_avail":90134364,"statfs":{"total":96636764160,"available":92297588736,"internally_reserved":0,"allocated":3514265600,"data_stored":4323404192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":824576662},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:18 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.68700000000000006,"15min":0.73799999999999999},"min":{"1min":0.20899999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":7.492},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.81299999999999994,"15min":0.85299999999999998},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.7250000000000001,"5min":6.2080000000000002,"15min":10.847},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:15:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.403,"15min":1.377},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.191},"max":{"1min":6.0919999999999996,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.601,"15min":1.2849999999999999},"min":{"1min":0.22800000000000001,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":4.8760000000000003,"5min":14.586,"15min":14.586},"last":0.96099999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739311,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10807736,"kb_used_data":9139628,"kb_used_omap":691,"kb_used_meta":1667404,"kb_avail":83564104,"statfs":{"total":96636764160,"available":85569642496,"internally_reserved":0,"allocated":9358979072,"data_stored":11514417722,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":1707422384},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,2,9,8,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":2.1110000000000002,"5min":0.84999999999999998,"15min":0.71799999999999997},"min":{"1min":0.21199999999999999,"5min":0.20100000000000001,"15min":0.189},"max":{"1min":23.434000000000001,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.313},{"interface":"front","average":{"1min":3.7949999999999999,"5min":1.1459999999999999,"15min":0.86299999999999999},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":24.908000000000001,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:15:51 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.304,"15min":1.1619999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":24.870000000000001,"5min":24.870000000000001,"15min":43.871000000000002},"last":7.149},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.601,"15min":1.3009999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":24.831,"5min":24.831,"15min":39.412999999999997},"last":6.9939999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739322,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6624340,"kb_used_data":5704524,"kb_used_omap":425,"kb_used_meta":919382,"kb_avail":87747500,"statfs":{"total":96636764160,"available":89853440000,"internally_reserved":0,"allocated":5841432576,"data_stored":7186898812,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":941447238},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.91000000000000003,"15min":0.86799999999999999},"min":{"1min":0.192,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.72399999999999998,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.376},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63400000000000001,"15min":0.59699999999999998},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":4.069,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.27300000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:15:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":0.81699999999999995,"15min":1.198},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":7.5030000000000001,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71399999999999997,"15min":0.80300000000000005},"min":{"1min":0.24399999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":2.7770000000000001,"5min":4.6630000000000003,"15min":8.9710000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5840261120,"data_stored":7186028154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9358430208,"data_stored":11514169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3512971264,"data_stored":4322943678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:15:57.389 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:57.389 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:57.390 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:57.390 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:57.390 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:57.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:57.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:57.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:57.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:57.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:15:57.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:57.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:15:57.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:57.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:57.818 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:57.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:58.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:58.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:58.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:58.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:58.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:58.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:58.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:58.229+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:15:58.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:58.269+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:58.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:58.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:58.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:58.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:15:59.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:59.621+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:15:59.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:59.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:15:59.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:15:59.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:00.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:00.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:16:00.067 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:00.065+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:16:00.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:00.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:00.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:00.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:00.101 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:00.101+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:00.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:00.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:00.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:00.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:00.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:00.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:00.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:00.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:00.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:00.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:00.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:00.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:02.026 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:02.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:02.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:02.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:16:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:02.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:02.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:02.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:02.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:02.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:02.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:02.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:02.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:03.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:03.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:03.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:03.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:03.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:03.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:03.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:03.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:04.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:04.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:06.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:06.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:06.767 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:06.764+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:06.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:06.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:06.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:06.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:06.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:06.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:06.772 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:06.768+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:07.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:06.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:16:07.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:06.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:07.005 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:06.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:07.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:07.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:08.729 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.728+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:08.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:08.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:08.749 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:08.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:16:08.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:08.980 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:08.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:08.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:08.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:08.990 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:08.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:09.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:09.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:09.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:09.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:09.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:09.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:09.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:09.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:09.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:09.976+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:09.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:09.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:10.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:10.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:16:10.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:10.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:10.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:10.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:10.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:10.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:11.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:11.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:16:11.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:11.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:11.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:11.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:11.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:11.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:11.404 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:11.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:11.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:11.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:16:11.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:11.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:11.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:11.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:11.922 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:11.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:11.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:11.924+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:11.930 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:11.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:11.931 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:11.928+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:12.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:12.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:12.378 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:12.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:12.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:12.376+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:13.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:13.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:13.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:13.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:13.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:13.616+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:13.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:13.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:15.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:16:15.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:16:15.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:16:15.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.560+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:16:15.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:16:15.568 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:15.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:15.590 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:15.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:15.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:15.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:15.612+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:15.616 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:15.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:15.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:15.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:15.630 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:15.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:15.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:15.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:15.631 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:15.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:15.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:15.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:15.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:15.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:15.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:17.392 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:18.080 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:16:18.081 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:16:18.115 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2548,"stamp":"2023-10-14T01:16:17.726293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28606909272,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2335445,"num_read_kb":9091867,"num_write":2293087,"num_write_kb":9172749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80594,"ondisk_log_size":80594,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21842480,"kb_used_data":18344148,"kb_used_omap":1434,"kb_used_meta":3496869,"kb_avail":261273040,"statfs":{"total":289910292480,"available":267543592960,"internally_reserved":0,"allocated":18784407552,"data_stored":23094451030,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1469044,"internal_metadata":3580794252},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,4,7,7,10,11,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17563648,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4097,"num_read_kb":16348,"num_write":4321,"num_write_kb":17284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"2.7","version":"146'307752","reported_seq":604185,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:15.502545+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:16:15.502545+0000","last_peered":"2023-10-14T01:16:15.502545+0000","last_clean":"2023-10-14T01:16:15.502545+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:16:15.502545+0000","last_undegraded":"2023-10-14T01:16:15.502545+0000","last_fullsized":"2023-10-14T01:16:15.502545+0000","mapping_epoch":13,"log_start":"143'297702","ondisk_log_start":"143'297702","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3618709942,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":297616,"num_read_kb":1153027,"num_write":289741,"num_write_kb":1158722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'307799","reported_seq":604838,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:15.467963+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:16:15.467963+0000","last_peered":"2023-10-14T01:16:15.467963+0000","last_clean":"2023-10-14T01:16:15.467963+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:16:15.467963+0000","last_undegraded":"2023-10-14T01:16:15.467963+0000","last_fullsized":"2023-10-14T01:16:15.467963+0000","mapping_epoch":13,"log_start":"143'297712","ondisk_log_start":"143'297712","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":662,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 347s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3610386800,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":299684,"num_read_kb":1151305,"num_write":289474,"num_write_kb":1157523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'278187","reported_seq":554174,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:15.424450+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:16:15.424450+0000","last_peered":"2023-10-14T01:16:15.424450+0000","last_clean":"2023-10-14T01:16:15.424450+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:16:15.424450+0000","last_undegraded":"2023-10-14T01:16:15.424450+0000","last_fullsized":"2023-10-14T01:16:15.424450+0000","mapping_epoch":13,"log_start":"143'268105","ondisk_log_start":"143'268105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3455312274,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":276408,"num_read_kb":1089792,"num_write":277569,"num_write_kb":1110246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'271853","reported_seq":544302,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:15.445268+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:16:15.445268+0000","last_peered":"2023-10-14T01:16:15.445268+0000","last_clean":"2023-10-14T01:16:15.445268+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:16:15.445268+0000","last_undegraded":"2023-10-14T01:16:15.445268+0000","last_fullsized":"2023-10-14T01:16:15.445268+0000","mapping_epoch":13,"log_start":"143'261807","ondisk_log_start":"143'261807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":644,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 226s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3380699267,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":272803,"num_read_kb":1073593,"num_write":270897,"num_write_kb":1083510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"146'306824","reported_seq":614556,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:16.071836+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:16:16.071836+0000","last_peered":"2023-10-14T01:16:16.071836+0000","last_clean":"2023-10-14T01:16:16.071836+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:16:16.071836+0000","last_undegraded":"2023-10-14T01:16:16.071836+0000","last_fullsized":"2023-10-14T01:16:16.071836+0000","mapping_epoch":13,"log_start":"143'296809","ondisk_log_start":"143'296809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3802771775,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":308140,"num_read_kb":1206818,"num_write":304407,"num_write_kb":1217481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"146'295795","reported_seq":586986,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:13.261319+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:16:13.261319+0000","last_peered":"2023-10-14T01:16:13.261319+0000","last_clean":"2023-10-14T01:16:13.261319+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:16:13.261319+0000","last_undegraded":"2023-10-14T01:16:13.261319+0000","last_fullsized":"2023-10-14T01:16:13.261319+0000","mapping_epoch":13,"log_start":"143'285709","ondisk_log_start":"143'285709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3567706381,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":292105,"num_read_kb":1131824,"num_write":286687,"num_write_kb":1146534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"146'300280","reported_seq":598911,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:13.248701+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:16:13.248701+0000","last_peered":"2023-10-14T01:16:13.248701+0000","last_clean":"2023-10-14T01:16:13.248701+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:16:13.248701+0000","last_undegraded":"2023-10-14T01:16:13.248701+0000","last_fullsized":"2023-10-14T01:16:13.248701+0000","mapping_epoch":13,"log_start":"143'290211","ondisk_log_start":"143'290211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3617116409,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":299699,"num_read_kb":1156064,"num_write":289266,"num_write_kb":1156790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"146'294741","reported_seq":583000,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:15.432640+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:16:15.432640+0000","last_peered":"2023-10-14T01:16:15.432640+0000","last_clean":"2023-10-14T01:16:15.432640+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:16:15.432640+0000","last_undegraded":"2023-10-14T01:16:15.432640+0000","last_fullsized":"2023-10-14T01:16:15.432640+0000","mapping_epoch":13,"log_start":"143'284709","ondisk_log_start":"143'284709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":885,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 354s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3553616056,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":288864,"num_read_kb":1129335,"num_write":284877,"num_write_kb":1139395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":356,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.689327+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:15:29.689327+0000","last_peered":"2023-10-14T01:15:29.689327+0000","last_clean":"2023-10-14T01:15:29.689327+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:15:29.689327+0000","last_undegraded":"2023-10-14T01:15:29.689327+0000","last_fullsized":"2023-10-14T01:15:29.689327+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28606318904,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2335319,"num_read_kb":9091758,"num_write":2292918,"num_write_kb":9170201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18781405184,"data_stored":23092884080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80467,"ondisk_log_size":80467,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739328,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4333736,"kb_used_data":3443488,"kb_used_omap":317,"kb_used_meta":889922,"kb_avail":90038104,"statfs":{"total":96636764160,"available":92199018496,"internally_reserved":0,"allocated":3526131712,"data_stored":4335270304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":911280790},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:18 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.68700000000000006,"15min":0.73799999999999999},"min":{"1min":0.20899999999999999,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":5.5830000000000002,"5min":5.5830000000000002,"15min":7.492},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.81299999999999994,"15min":0.85299999999999998},"min":{"1min":0.19900000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.7250000000000001,"5min":6.2080000000000002,"15min":10.847},"last":0.308}]},{"osd":1,"last update":"Sat Oct 14 01:15:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.403,"15min":1.377},"min":{"1min":0.23400000000000001,"5min":0.193,"15min":0.191},"max":{"1min":6.0919999999999996,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.85799999999999998,"5min":1.601,"15min":1.2849999999999999},"min":{"1min":0.22800000000000001,"5min":0.20899999999999999,"15min":0.20699999999999999},"max":{"1min":4.8760000000000003,"5min":14.586,"15min":14.586},"last":0.39100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739315,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10946932,"kb_used_data":9171944,"kb_used_omap":691,"kb_used_meta":1774284,"kb_avail":83424908,"statfs":{"total":96636764160,"available":85427105792,"internally_reserved":0,"allocated":9392070656,"data_stored":11547509306,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":1816867504},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,5,2,6,6,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":2.1110000000000002,"5min":0.84999999999999998,"15min":0.71799999999999997},"min":{"1min":0.21199999999999999,"5min":0.20100000000000001,"15min":0.189},"max":{"1min":23.434000000000001,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.7949999999999999,"5min":1.1459999999999999,"15min":0.86299999999999999},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":24.908000000000001,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:15:51 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.304,"15min":1.1619999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":24.870000000000001,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.601,"15min":1.3009999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":24.831,"5min":24.831,"15min":39.412999999999997},"last":0.30399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739327,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6561812,"kb_used_data":5728716,"kb_used_omap":425,"kb_used_meta":832662,"kb_avail":87810028,"statfs":{"total":96636764160,"available":89917468672,"internally_reserved":0,"allocated":5866205184,"data_stored":7211671420,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":852645958},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.91000000000000003,"15min":0.86799999999999999},"min":{"1min":0.192,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.72399999999999998,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63400000000000001,"15min":0.59699999999999998},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":4.069,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.376}]},{"osd":2,"last update":"Sat Oct 14 01:15:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":0.81699999999999995,"15min":1.198},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":7.5030000000000001,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71399999999999997,"15min":0.80300000000000005},"min":{"1min":0.24399999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":2.7770000000000001,"5min":4.6630000000000003,"15min":8.9710000000000001},"last":0.32200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5865385984,"data_stored":7211153018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9391144960,"data_stored":11546884408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3524874240,"data_stored":4334846654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:16:18.116 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:18.116 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:18.117 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:18.117 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:18.117 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:18.118 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:16:20.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:20.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:20.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:20.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:20.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:20.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:22.971 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:22.968+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:16:22.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:22.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:22.978 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:22.976+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:22.999 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:22.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:23.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:23.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:23.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:23.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:23.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:23.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:23.163 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:23.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:16:23.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:23.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:23.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:23.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:23.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:23.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:23.253 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:23.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:23.265 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:23.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:25.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:25.132+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:16:25.150 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:25.148+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:25.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:25.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:25.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:25.176+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:25.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:25.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:25.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:25.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:25.190 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:25.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:26.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:16:26.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:16:26.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:16:26.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:26.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:26.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:26.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:26.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:26.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:26.235 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:26.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:27.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:27.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:27.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:27.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:28.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:28.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:16:28.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:28.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:28.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:28.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:28.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:28.364+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:28.473 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:28.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:28.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:28.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:28.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:28.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:28.474 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:28.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:28.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:28.384+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:28.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:28.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:28.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:28.836+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:28.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:28.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:29.471 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:29.468+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:29.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:29.484+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:29.485 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:29.484+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:30.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:30.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:30.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:30.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:30.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:30.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:30.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:30.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:31.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:31.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:31.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:31.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:31.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:31.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:31.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:31.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:31.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:31.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:31.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:31.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:31.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:31.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:33.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:33.544+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:33.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:33.564+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:33.624 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:33.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:33.625 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:33.624+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:35.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:35.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:35.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:35.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:35.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:35.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:35.357 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:35.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:36.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:36.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:36.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:36.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:38.119 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:38.964 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:16:38.965 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:16:38.995 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2558,"stamp":"2023-10-14T01:16:37.729641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28639656792,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2343208,"num_read_kb":9122831,"num_write":2301102,"num_write_kb":9204809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80599,"ondisk_log_size":80599,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22168900,"kb_used_data":18408104,"kb_used_omap":1434,"kb_used_meta":3759333,"kb_avail":260946620,"statfs":{"total":289910292480,"available":267209338880,"internally_reserved":0,"allocated":18849898496,"data_stored":23159941974,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1469044,"internal_metadata":3849557388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,6,5,16,13,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22364160,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5316,"num_read_kb":21212,"num_write":5476,"num_write_kb":21904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002056"},"pg_stats":[{"pgid":"2.7","version":"146'308818","reported_seq":606270,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:36.124438+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:16:36.124438+0000","last_peered":"2023-10-14T01:16:36.124438+0000","last_clean":"2023-10-14T01:16:36.124438+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:16:36.124438+0000","last_undegraded":"2023-10-14T01:16:36.124438+0000","last_fullsized":"2023-10-14T01:16:36.124438+0000","mapping_epoch":13,"log_start":"144'298802","ondisk_log_start":"144'298802","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3622879670,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":298617,"num_read_kb":1157027,"num_write":290759,"num_write_kb":1162794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"146'308891","reported_seq":606940,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:36.129219+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:16:36.129219+0000","last_peered":"2023-10-14T01:16:36.129219+0000","last_clean":"2023-10-14T01:16:36.129219+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:16:36.129219+0000","last_undegraded":"2023-10-14T01:16:36.129219+0000","last_fullsized":"2023-10-14T01:16:36.129219+0000","mapping_epoch":13,"log_start":"144'298812","ondisk_log_start":"144'298812","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":701,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 368s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3614548336,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":300670,"num_read_kb":1155245,"num_write":290493,"num_write_kb":1161599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"146'279191","reported_seq":556099,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:36.087279+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:16:36.087279+0000","last_peered":"2023-10-14T01:16:36.087279+0000","last_clean":"2023-10-14T01:16:36.087279+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:16:36.087279+0000","last_undegraded":"2023-10-14T01:16:36.087279+0000","last_fullsized":"2023-10-14T01:16:36.087279+0000","mapping_epoch":13,"log_start":"143'269105","ondisk_log_start":"143'269105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3459428754,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":277330,"num_read_kb":1093464,"num_write":278573,"num_write_kb":1114262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'272838","reported_seq":546260,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:36.095925+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:16:36.095925+0000","last_peered":"2023-10-14T01:16:36.095925+0000","last_clean":"2023-10-14T01:16:36.095925+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:16:36.095925+0000","last_undegraded":"2023-10-14T01:16:36.095925+0000","last_fullsized":"2023-10-14T01:16:36.095925+0000","mapping_epoch":13,"log_start":"143'262807","ondisk_log_start":"143'262807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":693,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 246s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3384594563,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":273768,"num_read_kb":1077429,"num_write":271854,"num_write_kb":1087338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"146'307904","reported_seq":616661,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:36.208530+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:16:36.208530+0000","last_peered":"2023-10-14T01:16:36.208530+0000","last_clean":"2023-10-14T01:16:36.208530+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:16:36.208530+0000","last_undegraded":"2023-10-14T01:16:36.208530+0000","last_fullsized":"2023-10-14T01:16:36.208530+0000","mapping_epoch":13,"log_start":"144'297809","ondisk_log_start":"144'297809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3807134015,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":309161,"num_read_kb":1210886,"num_write":305482,"num_write_kb":1221781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"146'296813","reported_seq":588972,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:33.627560+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:16:33.627560+0000","last_peered":"2023-10-14T01:16:33.627560+0000","last_clean":"2023-10-14T01:16:33.627560+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:16:33.627560+0000","last_undegraded":"2023-10-14T01:16:33.627560+0000","last_fullsized":"2023-10-14T01:16:33.627560+0000","mapping_epoch":13,"log_start":"144'286809","ondisk_log_start":"144'286809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3571720461,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":293057,"num_read_kb":1135624,"num_write":287666,"num_write_kb":1150450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"146'301296","reported_seq":600892,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:16:33.625323+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:16:33.625323+0000","last_peered":"2023-10-14T01:16:33.625323+0000","last_clean":"2023-10-14T01:16:33.625323+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:16:33.625323+0000","last_undegraded":"2023-10-14T01:16:33.625323+0000","last_fullsized":"2023-10-14T01:16:33.625323+0000","mapping_epoch":13,"log_start":"144'291211","ondisk_log_start":"144'291211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3621060857,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":300642,"num_read_kb":1159824,"num_write":290234,"num_write_kb":1160662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"146'295785","reported_seq":585033,"reported_epoch":146,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:36.075565+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:16:36.075565+0000","last_peered":"2023-10-14T01:16:36.075565+0000","last_clean":"2023-10-14T01:16:36.075565+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:16:36.075565+0000","last_undegraded":"2023-10-14T01:16:36.075565+0000","last_fullsized":"2023-10-14T01:16:36.075565+0000","mapping_epoch":13,"log_start":"144'285709","ondisk_log_start":"144'285709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":929,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 374s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3557699768,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":289837,"num_read_kb":1133223,"num_write":285872,"num_write_kb":1143375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":356,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.689327+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:15:29.689327+0000","last_peered":"2023-10-14T01:15:29.689327+0000","last_clean":"2023-10-14T01:15:29.689327+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:15:29.689327+0000","last_undegraded":"2023-10-14T01:15:29.689327+0000","last_fullsized":"2023-10-14T01:15:29.689327+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28639066424,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2343082,"num_read_kb":9122722,"num_write":2300933,"num_write_kb":9202261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18846842880,"data_stored":23158321776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80472,"ondisk_log_size":80472,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739332,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4359044,"kb_used_data":3454908,"kb_used_omap":317,"kb_used_meta":903810,"kb_avail":90012796,"statfs":{"total":96636764160,"available":92173103104,"internally_reserved":0,"allocated":3537825792,"data_stored":4346964384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":925502102},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:20 2023","interfaces":[{"interface":"back","average":{"1min":1.8680000000000001,"5min":0.91700000000000004,"15min":0.83399999999999996},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":7.6440000000000001,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":0.83099999999999996,"15min":0.82799999999999996},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.6680000000000001,"5min":7.6680000000000001,"15min":7.782},"last":0.40400000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:16:20 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.4630000000000001,"15min":1.337},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":6.6299999999999999,"5min":11.077999999999999,"15min":14.103999999999999},"last":7.5199999999999996},{"interface":"front","average":{"1min":1.806,"5min":1.575,"15min":1.3109999999999999},"min":{"1min":0.249,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":5.3120000000000003,"5min":14.586,"15min":14.586},"last":7.1840000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739319,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11126028,"kb_used_data":9204548,"kb_used_omap":691,"kb_used_meta":1920780,"kb_avail":83245812,"statfs":{"total":96636764160,"available":85243711488,"internally_reserved":0,"allocated":9425457152,"data_stored":11580895802,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":1966879408},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,11,8,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:35 2023","interfaces":[{"interface":"back","average":{"1min":2.1110000000000002,"5min":0.84999999999999998,"15min":0.71799999999999997},"min":{"1min":0.21199999999999999,"5min":0.20100000000000001,"15min":0.189},"max":{"1min":23.434000000000001,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":3.7949999999999999,"5min":1.1459999999999999,"15min":0.86299999999999999},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":24.908000000000001,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:15:51 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.304,"15min":1.1619999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":24.870000000000001,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.601,"15min":1.3009999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":24.831,"5min":24.831,"15min":39.412999999999997},"last":0.30599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739331,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6683828,"kb_used_data":5748648,"kb_used_omap":425,"kb_used_meta":934742,"kb_avail":87688012,"statfs":{"total":96636764160,"available":89792524288,"internally_reserved":0,"allocated":5886615552,"data_stored":7232081788,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":957175878},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:39 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.91000000000000003,"15min":0.86799999999999999},"min":{"1min":0.192,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.72399999999999998,"5min":9.5380000000000003,"15min":9.5380000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63400000000000001,"15min":0.59699999999999998},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":4.069,"5min":5.7670000000000003,"15min":9.2439999999999998},"last":0.22}]},{"osd":2,"last update":"Sat Oct 14 01:15:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":0.81699999999999995,"15min":1.198},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":7.5030000000000001,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.246},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.71399999999999997,"15min":0.80300000000000005},"min":{"1min":0.24399999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":2.7770000000000001,"5min":4.6630000000000003,"15min":8.9710000000000001},"last":0.29999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5885779968,"data_stored":7231547002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9424138240,"data_stored":11579877688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3536924672,"data_stored":4346897086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:16:38.997 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:38.997 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:38.998 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:38.998 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:38.998 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:38.999 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:16:39.025 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:39.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:39.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:39.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:39.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:39.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:39.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:39.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:40.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:40.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:16:40.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:40.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:40.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:40.156+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:40.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:40.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:40.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:40.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:40.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:40.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:42.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:42.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:16:42.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:42.384+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:42.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:42.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:42.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:42.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:43.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:43.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:16:43.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:43.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:43.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:43.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:43.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:43.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:43.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:43.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:43.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:43.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:44.429 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:44.428+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:44.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:44.456+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:47.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:47.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:47.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:47.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:47.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:47.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:47.339 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:47.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:47.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:47.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:47.355 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:47.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:47.355 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:47.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:47.355 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:47.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:47.355 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:47.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:47.862 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:47.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:47.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:47.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:47.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:47.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:47.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:47.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:48.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:48.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:48.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:48.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:48.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:16:48.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:16:48.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:16:48.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:48.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:48.546 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:48.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:50.819 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:50.816+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:50.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:50.824+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:50.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:50.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:50.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:50.831 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:50.828+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:52.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:52.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:16:52.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:52.800+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:52.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:52.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:52.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:52.820+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:53.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:53.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:16:53.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:53.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:16:53.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:53.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:57.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:16:57.696+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:16:57.878 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:16:57.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:16:57.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:16:57.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:57.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:16:57.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:57.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:16:57.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:16:57.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:16:59.000 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:59.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:59.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:16:59.779 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:59.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:16:59.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:16:59.791+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:16:59.902 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:16:59.903 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:16:59.959 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2570,"stamp":"2023-10-14T01:16:59.737594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28674968408,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2351607,"num_read_kb":9156323,"num_write":2309747,"num_write_kb":9239389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80550,"ondisk_log_size":80550,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22372364,"kb_used_data":18475380,"kb_used_omap":1434,"kb_used_meta":3895525,"kb_avail":260743156,"statfs":{"total":289910292480,"available":267000991744,"internally_reserved":0,"allocated":18918789120,"data_stored":23228820901,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1469044,"internal_metadata":3989017996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,6,11,9,16,21,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18911232,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4502,"num_read_kb":17956,"num_write":4627,"num_write_kb":18508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003611"},"pg_stats":[{"pgid":"2.7","version":"147'309905","reported_seq":608444,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:16:56.857892+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:16:56.857892+0000","last_peered":"2023-10-14T01:16:56.857892+0000","last_clean":"2023-10-14T01:16:56.857892+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:16:56.857892+0000","last_undegraded":"2023-10-14T01:16:56.857892+0000","last_fullsized":"2023-10-14T01:16:56.857892+0000","mapping_epoch":13,"log_start":"144'299902","ondisk_log_start":"144'299902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3627147702,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":299688,"num_read_kb":1161287,"num_write":291801,"num_write_kb":1166962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'309993","reported_seq":609155,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:56.870435+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:16:56.870435+0000","last_peered":"2023-10-14T01:16:56.870435+0000","last_clean":"2023-10-14T01:16:56.870435+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:16:56.870435+0000","last_undegraded":"2023-10-14T01:16:56.870435+0000","last_fullsized":"2023-10-14T01:16:56.870435+0000","mapping_epoch":13,"log_start":"144'299912","ondisk_log_start":"144'299912","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":717,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 388s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3618845040,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":301758,"num_read_kb":1159585,"num_write":291543,"num_write_kb":1165799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'280189","reported_seq":558049,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:16:56.834273+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:16:56.834273+0000","last_peered":"2023-10-14T01:16:56.834273+0000","last_clean":"2023-10-14T01:16:56.834273+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:16:56.834273+0000","last_undegraded":"2023-10-14T01:16:56.834273+0000","last_fullsized":"2023-10-14T01:16:56.834273+0000","mapping_epoch":13,"log_start":"144'270105","ondisk_log_start":"144'270105","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3463512466,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":278283,"num_read_kb":1097268,"num_write":279571,"num_write_kb":1118254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'273868","reported_seq":548191,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:56.863185+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:16:56.863185+0000","last_peered":"2023-10-14T01:16:56.863185+0000","last_clean":"2023-10-14T01:16:56.863185+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:16:56.863185+0000","last_undegraded":"2023-10-14T01:16:56.863185+0000","last_fullsized":"2023-10-14T01:16:56.863185+0000","mapping_epoch":13,"log_start":"144'263807","ondisk_log_start":"144'263807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":743,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 267s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3388694659,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":274662,"num_read_kb":1080997,"num_write":272855,"num_write_kb":1091342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"147'308930","reported_seq":618741,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:16:56.389139+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:16:56.389139+0000","last_peered":"2023-10-14T01:16:56.389139+0000","last_clean":"2023-10-14T01:16:56.389139+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:16:56.389139+0000","last_undegraded":"2023-10-14T01:16:56.389139+0000","last_fullsized":"2023-10-14T01:16:56.389139+0000","mapping_epoch":13,"log_start":"144'298909","ondisk_log_start":"144'298909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3811311935,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":310208,"num_read_kb":1215062,"num_write":306505,"num_write_kb":1225873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"147'298050","reported_seq":591422,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:16:58.761299+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:16:58.761299+0000","last_peered":"2023-10-14T01:16:58.761299+0000","last_clean":"2023-10-14T01:16:58.761299+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:16:58.761299+0000","last_undegraded":"2023-10-14T01:16:58.761299+0000","last_fullsized":"2023-10-14T01:16:58.761299+0000","mapping_epoch":13,"log_start":"144'288009","ondisk_log_start":"144'288009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3576676621,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":294246,"num_read_kb":1140376,"num_write":288883,"num_write_kb":1155318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"147'302570","reported_seq":603334,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:16:58.765260+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:16:58.765260+0000","last_peered":"2023-10-14T01:16:58.765260+0000","last_clean":"2023-10-14T01:16:58.765260+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:16:58.765260+0000","last_undegraded":"2023-10-14T01:16:58.765260+0000","last_fullsized":"2023-10-14T01:16:58.765260+0000","mapping_epoch":13,"log_start":"144'292511","ondisk_log_start":"144'292511","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3626143993,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":301787,"num_read_kb":1164380,"num_write":291482,"num_write_kb":1165654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"147'296882","reported_seq":587156,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:56.838888+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:16:56.838888+0000","last_peered":"2023-10-14T01:16:56.838888+0000","last_clean":"2023-10-14T01:16:56.838888+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:16:56.838888+0000","last_undegraded":"2023-10-14T01:16:56.838888+0000","last_fullsized":"2023-10-14T01:16:56.838888+0000","mapping_epoch":13,"log_start":"144'286809","ondisk_log_start":"144'286809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":947,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 395s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3562045624,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":290849,"num_read_kb":1137259,"num_write":286938,"num_write_kb":1147639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":357,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:54.934188+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:15:54.934188+0000","last_peered":"2023-10-14T01:15:54.934188+0000","last_clean":"2023-10-14T01:15:54.934188+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:15:54.934188+0000","last_undegraded":"2023-10-14T01:15:54.934188+0000","last_fullsized":"2023-10-14T01:15:54.934188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28674378040,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2351481,"num_read_kb":9156214,"num_write":2309578,"num_write_kb":9236841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18916241408,"data_stored":23227720304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80423,"ondisk_log_size":80423,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739337,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4428536,"kb_used_data":3470256,"kb_used_omap":317,"kb_used_meta":957954,"kb_avail":89943304,"statfs":{"total":96636764160,"available":92101943296,"internally_reserved":0,"allocated":3553542144,"data_stored":4362676837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":980945558},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:20 2023","interfaces":[{"interface":"back","average":{"1min":1.8680000000000001,"5min":0.91700000000000004,"15min":0.83399999999999996},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":7.6440000000000001,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":0.83099999999999996,"15min":0.82799999999999996},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.6680000000000001,"5min":7.6680000000000001,"15min":7.782},"last":0.498}]},{"osd":1,"last update":"Sat Oct 14 01:16:20 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.4630000000000001,"15min":1.337},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":6.6299999999999999,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.806,"5min":1.575,"15min":1.3109999999999999},"min":{"1min":0.249,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":5.3120000000000003,"5min":14.586,"15min":14.586},"last":0.57599999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739323,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11288564,"kb_used_data":9236588,"kb_used_omap":691,"kb_used_meta":2051276,"kb_avail":83083276,"statfs":{"total":96636764160,"available":85077274624,"internally_reserved":0,"allocated":9458266112,"data_stored":11613700863,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":707920,"internal_metadata":2100507312},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,3,5,7,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.96499999999999997,"15min":0.72699999999999998},"min":{"1min":0.23499999999999999,"5min":0.20100000000000001,"15min":0.19400000000000001},"max":{"1min":7.4219999999999997,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.044,"5min":1.2649999999999999,"15min":0.871},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":7.3529999999999998,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:15:51 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.304,"15min":1.1619999999999999},"min":{"1min":0.23899999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":24.870000000000001,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.601,"15min":1.3009999999999999},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":24.831,"5min":24.831,"15min":39.412999999999997},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359739335,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6655264,"kb_used_data":5768536,"kb_used_omap":425,"kb_used_meta":886294,"kb_avail":87716576,"statfs":{"total":96636764160,"available":89821773824,"internally_reserved":0,"allocated":5906980864,"data_stored":7252443201,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":907565126},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":0.99199999999999999,"15min":0.89000000000000001},"min":{"1min":0.188,"5min":0.186,"15min":0.17599999999999999},"max":{"1min":6.1180000000000003,"5min":6.952,"15min":9.5380000000000003},"last":3.1000000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.61499999999999999,"15min":0.59499999999999997},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":2.9849999999999999,"5min":5.351,"15min":9.2439999999999998},"last":3.0649999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:16:43 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":0.94899999999999995,"15min":1.2190000000000001},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":6.3029999999999999,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.253},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.68000000000000005,"15min":0.82199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5906247680,"data_stored":7252014714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9457774592,"data_stored":11613514040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3552219136,"data_stored":4362191550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:16:59.961 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:59.961 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:59.962 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:59.962 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:59.963 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:59.963 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:17:00.746 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:17:00.806 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-14T01:17:01.532 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-14T01:17:01.612 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:17:02.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:17:02.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.039+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:17:02.052 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:17:02.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:02.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:02.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.127+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:02.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:02.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:02.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:02.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:02.454 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:17:02.513 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-14T01:17:03.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:03.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:03.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:03.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:03.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:03.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:03.401 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-14T01:17:03.456 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-14T01:17:03.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:03.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:17:03.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:03.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:03.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:03.895+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:04.233 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-14T01:17:04.282 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:04.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:04.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:04.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:04.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:04.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:04.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:04.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:04.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:04.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:04.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:04.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:04.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:04.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:04.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:05.045 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.043+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:05.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.075+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:05.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.079+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:05.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.095+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:05.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.095+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:17:05.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.095+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:17:05.132 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.127+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:17:05.132 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.127+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:05.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:05.143 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:05.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:05.155+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:05.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:05.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:05.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:05.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:06.299 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:06.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:17:06.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:06.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:06.313 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:06.311+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:06.324 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:06.323+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:06.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:06.347+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:06.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:06.347+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:17:06.352 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:06.347+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:17:07.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:07.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:07.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:07.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:07.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:07.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:07.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:07.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:07.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:07.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:07.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.879+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:17:07.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:17:07.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:07.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:07.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:07.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:07.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:17:07.975 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:07.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:17:08.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:08.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:08.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:08.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:08.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:08.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:09.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:09.807+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:17:09.809 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:09.807+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:09.810 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:09.807+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:09.816 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:09.815+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:09.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:09.823+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:09.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:09.835+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:17:09.852 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:09.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:17:11.603 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:11.599+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:11.628 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:11.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:11.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:11.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:11.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:11.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:11.654 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:11.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:11.674 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:11.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:11.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:11.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:11.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:11.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:17:11.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:11.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:17:14.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:14.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:14.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:14.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:15.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:15.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:15.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:15.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:15.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:15.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:15.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:15.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:16.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:16.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:16.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:16.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:16.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:16.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:16.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:16.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:16.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:16.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:16.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:16.279+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:17.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:17.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:17.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:17.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:17.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:17.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:17.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:17.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:20.256 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:20.255+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:17:20.256 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:20.255+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:20.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:20.279+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:20.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:20.295+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:20.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:20.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:17:20.311 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:20.307+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:17:21.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:21.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:21.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:21.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:21.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:21.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:21.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:21.335+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:21.341 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:21.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:24.283 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:24.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:24.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:24.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:24.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:24.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:24.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:24.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:24.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:24.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:24.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:24.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:24.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:25.082 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:17:25.082 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:17:25.154 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2582,"stamp":"2023-10-14T01:17:23.743414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28712323928,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2360795,"num_read_kb":9192995,"num_write":2318921,"num_write_kb":9276085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80475,"ondisk_log_size":80475,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22154956,"kb_used_data":18548916,"kb_used_omap":1500,"kb_used_meta":3604515,"kb_avail":260960564,"statfs":{"total":289910292480,"available":267223617536,"internally_reserved":0,"allocated":18994089984,"data_stored":23304121765,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1536928,"internal_metadata":3691023456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,5,3,6,7,19,18,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20598784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5092,"num_read_kb":20308,"num_write":5073,"num_write_kb":20292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003693"},"pg_stats":[{"pgid":"2.7","version":"147'311181","reported_seq":611004,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:22.629971+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:17:22.629971+0000","last_peered":"2023-10-14T01:17:22.629971+0000","last_clean":"2023-10-14T01:17:22.629971+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:17:22.629971+0000","last_undegraded":"2023-10-14T01:17:22.629971+0000","last_fullsized":"2023-10-14T01:17:22.629971+0000","mapping_epoch":13,"log_start":"144'301102","ondisk_log_start":"144'301102","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3632120246,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":300951,"num_read_kb":1166335,"num_write":293017,"num_write_kb":1171826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'311282","reported_seq":611711,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:22.605922+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:17:22.605922+0000","last_peered":"2023-10-14T01:17:22.605922+0000","last_clean":"2023-10-14T01:17:22.605922+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:17:22.605922+0000","last_undegraded":"2023-10-14T01:17:22.605922+0000","last_fullsized":"2023-10-14T01:17:22.605922+0000","mapping_epoch":13,"log_start":"144'301212","ondisk_log_start":"144'301212","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":768,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 414s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3623735664,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":302993,"num_read_kb":1164517,"num_write":292743,"num_write_kb":1170599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'281383","reported_seq":560372,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:22.599243+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:17:22.599243+0000","last_peered":"2023-10-14T01:17:22.599243+0000","last_clean":"2023-10-14T01:17:22.599243+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:17:22.599243+0000","last_undegraded":"2023-10-14T01:17:22.599243+0000","last_fullsized":"2023-10-14T01:17:22.599243+0000","mapping_epoch":13,"log_start":"144'271305","ondisk_log_start":"144'271305","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3468362130,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":279412,"num_read_kb":1101772,"num_write":280765,"num_write_kb":1123030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'275022","reported_seq":550525,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:22.575782+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:17:22.575782+0000","last_peered":"2023-10-14T01:17:22.575782+0000","last_clean":"2023-10-14T01:17:22.575782+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:17:22.575782+0000","last_undegraded":"2023-10-14T01:17:22.575782+0000","last_fullsized":"2023-10-14T01:17:22.575782+0000","mapping_epoch":13,"log_start":"144'265007","ondisk_log_start":"144'265007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":831,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 293s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3393175683,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":275826,"num_read_kb":1085653,"num_write":273956,"num_write_kb":1095746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"147'310231","reported_seq":621374,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:21.675290+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:17:21.675290+0000","last_peered":"2023-10-14T01:17:21.675290+0000","last_clean":"2023-10-14T01:17:21.675290+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:17:21.675290+0000","last_undegraded":"2023-10-14T01:17:21.675290+0000","last_fullsized":"2023-10-14T01:17:21.675290+0000","mapping_epoch":13,"log_start":"144'300209","ondisk_log_start":"144'300209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3816591679,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":311536,"num_read_kb":1220366,"num_write":307798,"num_write_kb":1231045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"147'299079","reported_seq":593395,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:18.958117+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:17:18.958117+0000","last_peered":"2023-10-14T01:17:18.958117+0000","last_clean":"2023-10-14T01:17:18.958117+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:17:18.958117+0000","last_undegraded":"2023-10-14T01:17:18.958117+0000","last_fullsized":"2023-10-14T01:17:18.958117+0000","mapping_epoch":13,"log_start":"144'289009","ondisk_log_start":"144'289009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3580690701,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":295174,"num_read_kb":1144080,"num_write":289876,"num_write_kb":1159290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"147'303612","reported_seq":605331,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:18.945290+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:17:18.945290+0000","last_peered":"2023-10-14T01:17:18.945290+0000","last_clean":"2023-10-14T01:17:18.945290+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:17:18.945290+0000","last_undegraded":"2023-10-14T01:17:18.945290+0000","last_fullsized":"2023-10-14T01:17:18.945290+0000","mapping_epoch":13,"log_start":"144'293611","ondisk_log_start":"144'293611","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3630211321,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":302715,"num_read_kb":1168064,"num_write":292479,"num_write_kb":1169642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"147'298122","reported_seq":589633,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:22.524497+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:17:22.524497+0000","last_peered":"2023-10-14T01:17:22.524497+0000","last_clean":"2023-10-14T01:17:22.524497+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:17:22.524497+0000","last_undegraded":"2023-10-14T01:17:22.524497+0000","last_fullsized":"2023-10-14T01:17:22.524497+0000","mapping_epoch":13,"log_start":"144'288109","ondisk_log_start":"144'288109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":995,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 421s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3566846136,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":292062,"num_read_kb":1142099,"num_write":288118,"num_write_kb":1152359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":357,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:54.934188+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:15:54.934188+0000","last_peered":"2023-10-14T01:15:54.934188+0000","last_clean":"2023-10-14T01:15:54.934188+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:15:54.934188+0000","last_undegraded":"2023-10-14T01:15:54.934188+0000","last_fullsized":"2023-10-14T01:15:54.934188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28711733560,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2360669,"num_read_kb":9192886,"num_write":2318752,"num_write_kb":9273537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18992508928,"data_stored":23303987824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80348,"ondisk_log_size":80348,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739341,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4495304,"kb_used_data":3481856,"kb_used_omap":317,"kb_used_meta":1013122,"kb_avail":89876536,"statfs":{"total":96636764160,"available":92033572864,"internally_reserved":0,"allocated":3565420544,"data_stored":4374555237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1037437590},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,1,2,0,8,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:20 2023","interfaces":[{"interface":"back","average":{"1min":1.8680000000000001,"5min":0.91700000000000004,"15min":0.83399999999999996},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":7.6440000000000001,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.435},{"interface":"front","average":{"1min":1.3240000000000001,"5min":0.83099999999999996,"15min":0.82799999999999996},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.6680000000000001,"5min":7.6680000000000001,"15min":7.782},"last":0.26700000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:16:20 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.4630000000000001,"15min":1.337},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.191},"max":{"1min":6.6299999999999999,"5min":11.077999999999999,"15min":14.103999999999999},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.806,"5min":1.575,"15min":1.3109999999999999},"min":{"1min":0.249,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":5.3120000000000003,"5min":14.586,"15min":14.586},"last":6.6040000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739328,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10885276,"kb_used_data":9275092,"kb_used_omap":757,"kb_used_meta":1609418,"kb_avail":83486564,"statfs":{"total":96636764160,"available":85490241536,"internally_reserved":0,"allocated":9497694208,"data_stored":11653128959,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":775804,"internal_metadata":1648044420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,4,5,8,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.96499999999999997,"15min":0.72699999999999998},"min":{"1min":0.23499999999999999,"5min":0.20100000000000001,"15min":0.19400000000000001},"max":{"1min":7.4219999999999997,"5min":23.434000000000001,"15min":23.434000000000001},"last":6.8739999999999997},{"interface":"front","average":{"1min":1.044,"5min":1.2649999999999999,"15min":0.871},"min":{"1min":0.254,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":7.3529999999999998,"5min":24.908000000000001,"15min":24.908000000000001},"last":6.915}]},{"osd":2,"last update":"Sat Oct 14 01:16:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1639999999999999,"15min":1.1779999999999999},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":7.149,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.448,"15min":1.321},"min":{"1min":0.218,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":6.9939999999999998,"5min":24.831,"15min":39.412999999999997},"last":5.4000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739340,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6774376,"kb_used_data":5791968,"kb_used_omap":425,"kb_used_meta":981974,"kb_avail":87597464,"statfs":{"total":96636764160,"available":89699803136,"internally_reserved":0,"allocated":5930975232,"data_stored":7276437569,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1005541446},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":0.99199999999999999,"15min":0.89000000000000001},"min":{"1min":0.188,"5min":0.186,"15min":0.17599999999999999},"max":{"1min":6.1180000000000003,"5min":6.952,"15min":9.5380000000000003},"last":2.7669999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.61499999999999999,"15min":0.59499999999999997},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":2.9849999999999999,"5min":5.351,"15min":9.2439999999999998},"last":0.317}]},{"osd":2,"last update":"Sat Oct 14 01:16:43 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":0.94899999999999995,"15min":1.2190000000000001},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":6.3029999999999999,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.68000000000000005,"15min":0.82199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5930471424,"data_stored":7276238458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9497804800,"data_stored":11653544248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3564232704,"data_stored":4374205118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:17:25.155 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:25.156 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:25.156 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:25.156 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:25.157 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:25.157 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:26.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:26.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:26.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:26.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:26.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:26.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:26.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:26.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:26.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:26.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:26.755 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:26.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:17:26.755 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:26.751+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:17:26.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:26.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:26.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:26.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:17:26.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:26.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:26.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:26.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:17:28.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:28.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:28.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:28.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:28.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:28.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:30.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:30.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:30.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:30.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:30.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:30.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:30.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:30.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:34.066 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:34.063+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:34.074 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:34.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:34.084 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:34.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:34.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:34.099+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:34.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:34.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:35.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:35.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:35.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:35.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:37.934 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:37.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:37.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:37.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:37.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:37.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:40.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:40.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:17:40.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:40.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:40.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:40.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:40.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:40.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:40.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:40.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:40.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:40.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:40.285 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:40.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:41.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:41.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:41.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:41.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:41.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:41.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:43.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:43.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:43.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:43.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:43.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:43.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:43.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:43.835+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:43.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:43.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:43.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:43.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:43.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:43.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:45.159 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:45.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:45.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:45.194 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:45.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:45.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.227+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:45.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:45.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.399+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:45.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:45.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:45.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:45.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:45.892 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:17:45.894 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:17:45.940 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2594,"stamp":"2023-10-14T01:17:45.753779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28746853208,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2369051,"num_read_kb":9225943,"num_write":2327403,"num_write_kb":9310013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80555,"ondisk_log_size":80555,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22415784,"kb_used_data":18616016,"kb_used_omap":1500,"kb_used_meta":3798243,"kb_avail":260699736,"statfs":{"total":289910292480,"available":266956529664,"internally_reserved":0,"allocated":19062800384,"data_stored":23372832165,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1536928,"internal_metadata":3889400928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,9,7,10,23,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16617472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3942,"num_read_kb":15728,"num_write":4078,"num_write_kb":16312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008297"},"pg_stats":[{"pgid":"2.7","version":"147'312254","reported_seq":613125,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:43.322314+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:17:43.322314+0000","last_peered":"2023-10-14T01:17:43.322314+0000","last_clean":"2023-10-14T01:17:43.322314+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:17:43.322314+0000","last_undegraded":"2023-10-14T01:17:43.322314+0000","last_fullsized":"2023-10-14T01:17:43.322314+0000","mapping_epoch":13,"log_start":"145'302202","ondisk_log_start":"145'302202","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3636335030,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":301982,"num_read_kb":1170439,"num_write":294053,"num_write_kb":1175970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"147'312368","reported_seq":613828,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:43.294168+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:17:43.294168+0000","last_peered":"2023-10-14T01:17:43.294168+0000","last_clean":"2023-10-14T01:17:43.294168+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:17:43.294168+0000","last_undegraded":"2023-10-14T01:17:43.294168+0000","last_fullsized":"2023-10-14T01:17:43.294168+0000","mapping_epoch":13,"log_start":"145'302312","ondisk_log_start":"145'302312","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":803,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 435s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3627917680,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":304002,"num_read_kb":1168549,"num_write":293772,"num_write_kb":1174715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"147'282309","reported_seq":562254,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:43.264351+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:17:43.264351+0000","last_peered":"2023-10-14T01:17:43.264351+0000","last_clean":"2023-10-14T01:17:43.264351+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:17:43.264351+0000","last_undegraded":"2023-10-14T01:17:43.264351+0000","last_fullsized":"2023-10-14T01:17:43.264351+0000","mapping_epoch":13,"log_start":"144'272305","ondisk_log_start":"144'272305","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3472101778,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":280365,"num_read_kb":1105576,"num_write":281691,"num_write_kb":1126734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"147'275979","reported_seq":552451,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:43.270664+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:17:43.270664+0000","last_peered":"2023-10-14T01:17:43.270664+0000","last_clean":"2023-10-14T01:17:43.270664+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:17:43.270664+0000","last_undegraded":"2023-10-14T01:17:43.270664+0000","last_fullsized":"2023-10-14T01:17:43.270664+0000","mapping_epoch":13,"log_start":"144'265907","ondisk_log_start":"144'265907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":869,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 314s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3397001347,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":276785,"num_read_kb":1089481,"num_write":274894,"num_write_kb":1099498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"147'311310","reported_seq":623448,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:42.015999+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:17:42.015999+0000","last_peered":"2023-10-14T01:17:42.015999+0000","last_clean":"2023-10-14T01:17:42.015999+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:17:42.015999+0000","last_undegraded":"2023-10-14T01:17:42.015999+0000","last_fullsized":"2023-10-14T01:17:42.015999+0000","mapping_epoch":13,"log_start":"145'301309","ondisk_log_start":"145'301309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3820945727,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":312528,"num_read_kb":1224326,"num_write":308871,"num_write_kb":1235337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"147'300304","reported_seq":595826,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:44.189812+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:17:44.189812+0000","last_peered":"2023-10-14T01:17:44.189812+0000","last_clean":"2023-10-14T01:17:44.189812+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:17:44.189812+0000","last_undegraded":"2023-10-14T01:17:44.189812+0000","last_fullsized":"2023-10-14T01:17:44.189812+0000","mapping_epoch":13,"log_start":"145'290209","ondisk_log_start":"145'290209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3585663245,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":296359,"num_read_kb":1148808,"num_write":291093,"num_write_kb":1164158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"147'304871","reported_seq":607790,"reported_epoch":147,"state":"active+clean","last_fresh":"2023-10-14T01:17:44.106007+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:17:44.106007+0000","last_peered":"2023-10-14T01:17:44.106007+0000","last_clean":"2023-10-14T01:17:44.106007+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:17:44.106007+0000","last_undegraded":"2023-10-14T01:17:44.106007+0000","last_fullsized":"2023-10-14T01:17:44.106007+0000","mapping_epoch":13,"log_start":"145'294811","ondisk_log_start":"145'294811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3635327225,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":303896,"num_read_kb":1172788,"num_write":293728,"num_write_kb":1174638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"147'299197","reported_seq":591671,"reported_epoch":147,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:43.190724+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:17:43.190724+0000","last_peered":"2023-10-14T01:17:43.190724+0000","last_clean":"2023-10-14T01:17:43.190724+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:17:43.190724+0000","last_undegraded":"2023-10-14T01:17:43.190724+0000","last_fullsized":"2023-10-14T01:17:43.190724+0000","mapping_epoch":13,"log_start":"145'289109","ondisk_log_start":"145'289109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1077,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 442s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3570970808,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":293008,"num_read_kb":1145867,"num_write":289132,"num_write_kb":1156415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":357,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-10-14T01:15:54.934188+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:15:54.934188+0000","last_peered":"2023-10-14T01:15:54.934188+0000","last_clean":"2023-10-14T01:15:54.934188+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:15:54.934188+0000","last_undegraded":"2023-10-14T01:15:54.934188+0000","last_fullsized":"2023-10-14T01:15:54.934188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28746262840,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2368925,"num_read_kb":9225834,"num_write":2327234,"num_write_kb":9307465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19060686848,"data_stored":23372165744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80428,"ondisk_log_size":80428,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739346,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4570080,"kb_used_data":3496216,"kb_used_omap":317,"kb_used_meta":1073538,"kb_avail":89801760,"statfs":{"total":96636764160,"available":91957002240,"internally_reserved":0,"allocated":3580125184,"data_stored":4389259877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1099303574},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:21 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.98599999999999999,"15min":0.82799999999999996},"min":{"1min":0.27300000000000002,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":5.1529999999999996,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.77000000000000002,"15min":0.82299999999999995},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.476,"5min":7.6680000000000001,"15min":7.782},"last":0.26400000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:17:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3859999999999999,"15min":1.3540000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.5199999999999996,"5min":10.726000000000001,"15min":14.103999999999999},"last":0.23799999999999999},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.6870000000000001,"15min":1.446},"min":{"1min":0.27100000000000002,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":8.8650000000000002,"5min":9.1920000000000002,"15min":14.586},"last":0.29599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739332,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10965868,"kb_used_data":9307364,"kb_used_omap":757,"kb_used_meta":1657738,"kb_avail":83405972,"statfs":{"total":96636764160,"available":85407715328,"internally_reserved":0,"allocated":9530740736,"data_stored":11686175487,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":775804,"internal_metadata":1697524100},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,6,5,15,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.0349999999999999,"15min":0.749},"min":{"1min":0.23999999999999999,"5min":0.20100000000000001,"15min":0.19400000000000001},"max":{"1min":6.8739999999999997,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.3759999999999999,"15min":0.90600000000000003},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.915,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:16:56 2023","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.1639999999999999,"15min":1.1779999999999999},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":7.149,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.448,"15min":1.321},"min":{"1min":0.218,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":6.9939999999999998,"5min":24.831,"15min":39.412999999999997},"last":0.31900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739344,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6879836,"kb_used_data":5812436,"kb_used_omap":425,"kb_used_meta":1066966,"kb_avail":87492004,"statfs":{"total":96636764160,"available":89591812096,"internally_reserved":0,"allocated":5951934464,"data_stored":7297396801,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1092573254},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,4,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":0.99199999999999999,"15min":0.89000000000000001},"min":{"1min":0.188,"5min":0.186,"15min":0.17599999999999999},"max":{"1min":6.1180000000000003,"5min":6.952,"15min":9.5380000000000003},"last":1.008},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.61499999999999999,"15min":0.59499999999999997},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":2.9849999999999999,"5min":5.351,"15min":9.2439999999999998},"last":0.98399999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:16:43 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":0.94899999999999995,"15min":1.2190000000000001},"min":{"1min":0.21299999999999999,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":6.3029999999999999,"5min":9.5809999999999995,"15min":10.539999999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.68000000000000005,"15min":0.82199999999999995},"min":{"1min":0.223,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":4.4260000000000002,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":1.0660000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5950664704,"data_stored":7296431738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9530753024,"data_stored":11686492472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3579269120,"data_stored":4389241534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:17:45.941 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:45.942 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:45.942 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:45.943 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:45.943 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:45.943 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:46.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:46.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:46.813 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:46.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:47.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:47.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:47.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:47.211+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:47.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:47.215+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:47.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:47.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:47.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:47.267+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:48.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:48.863+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:48.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:48.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:48.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:48.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:48.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:48.935+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:50.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:50.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:50.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:50.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:50.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:50.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:50.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:50.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:50.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:50.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:50.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:51.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:17:51.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:51.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:51.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:51.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:51.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.894+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:51.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:51.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:51.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:54.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:54.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:54.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:54.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:55.062 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:55.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:55.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:55.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:55.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:55.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:55.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:55.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:55.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:55.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:55.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:55.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:56.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:17:56.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:17:56.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:17:56.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:17:56.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:17:56.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:56.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:56.106 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:56.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:58.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:58.510+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:17:58.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:58.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:17:58.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:17:58.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:17:58.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:58.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:17:58.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:58.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:17:58.855 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:58.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:17:58.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:17:58.850+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:02.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:02.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:02.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:02.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:02.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:02.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:03.273 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:03.270+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:03.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:03.270+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:04.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:04.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:04.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:04.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:04.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:04.202+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:04.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:04.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:04.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:04.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:04.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:04.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:05.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:05.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:05.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:05.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:05.412 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:05.406+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:05.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:05.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:05.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:05.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:05.945 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:06.771 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:18:06.771 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:18:06.828 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2605,"stamp":"2023-10-14T01:18:05.757237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28779043672,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2376954,"num_read_kb":9257439,"num_write":2335313,"num_write_kb":9341653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80453,"ondisk_log_size":80453,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22648360,"kb_used_data":18680648,"kb_used_omap":1500,"kb_used_meta":3966179,"kb_avail":260467160,"statfs":{"total":289910292480,"available":266718371840,"internally_reserved":0,"allocated":19128983552,"data_stored":23438991552,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1536928,"internal_metadata":4061367392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,8,11,11,14,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15974400,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3853,"num_read_kb":15352,"num_write":3922,"num_write_kb":15688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001615"},"pg_stats":[{"pgid":"2.7","version":"149'313336","reported_seq":615275,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:03.741827+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:18:03.741827+0000","last_peered":"2023-10-14T01:18:03.741827+0000","last_clean":"2023-10-14T01:18:03.741827+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:18:03.741827+0000","last_undegraded":"2023-10-14T01:18:03.741827+0000","last_fullsized":"2023-10-14T01:18:03.741827+0000","mapping_epoch":13,"log_start":"145'303302","ondisk_log_start":"145'303302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3640443318,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":303032,"num_read_kb":1174631,"num_write":295067,"num_write_kb":1180026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'313451","reported_seq":615913,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:03.746026+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:18:03.746026+0000","last_peered":"2023-10-14T01:18:03.746026+0000","last_clean":"2023-10-14T01:18:03.746026+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:18:03.746026+0000","last_undegraded":"2023-10-14T01:18:03.746026+0000","last_fullsized":"2023-10-14T01:18:03.746026+0000","mapping_epoch":13,"log_start":"145'303412","ondisk_log_start":"145'303412","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":871,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 455s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3631915376,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":304977,"num_read_kb":1172433,"num_write":294750,"num_write_kb":1178627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'283294","reported_seq":564186,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:03.734047+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:18:03.734047+0000","last_peered":"2023-10-14T01:18:03.734047+0000","last_clean":"2023-10-14T01:18:03.734047+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:18:03.734047+0000","last_undegraded":"2023-10-14T01:18:03.734047+0000","last_fullsized":"2023-10-14T01:18:03.734047+0000","mapping_epoch":13,"log_start":"145'273205","ondisk_log_start":"145'273205","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3476107666,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":281309,"num_read_kb":1109328,"num_write":282676,"num_write_kb":1130674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'276984","reported_seq":554378,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:03.745181+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:18:03.745181+0000","last_peered":"2023-10-14T01:18:03.745181+0000","last_clean":"2023-10-14T01:18:03.745181+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:18:03.745181+0000","last_undegraded":"2023-10-14T01:18:03.745181+0000","last_fullsized":"2023-10-14T01:18:03.745181+0000","mapping_epoch":13,"log_start":"145'266907","ondisk_log_start":"145'266907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":924,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 334s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3400970371,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":277696,"num_read_kb":1093113,"num_write":275869,"num_write_kb":1103398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"149'312354","reported_seq":625527,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:02.189220+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:18:02.189220+0000","last_peered":"2023-10-14T01:18:02.189220+0000","last_clean":"2023-10-14T01:18:02.189220+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:18:02.189220+0000","last_undegraded":"2023-10-14T01:18:02.189220+0000","last_fullsized":"2023-10-14T01:18:02.189220+0000","mapping_epoch":13,"log_start":"145'302309","ondisk_log_start":"145'302309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3825156415,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":313556,"num_read_kb":1228418,"num_write":309910,"num_write_kb":1239493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"149'301320","reported_seq":597903,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:04.461579+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:18:04.461579+0000","last_peered":"2023-10-14T01:18:04.461579+0000","last_clean":"2023-10-14T01:18:04.461579+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:18:04.461579+0000","last_undegraded":"2023-10-14T01:18:04.461579+0000","last_fullsized":"2023-10-14T01:18:04.461579+0000","mapping_epoch":13,"log_start":"145'291309","ondisk_log_start":"145'291309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3589751053,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":297406,"num_read_kb":1152984,"num_write":292097,"num_write_kb":1168174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"149'305830","reported_seq":609703,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:04.475875+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:18:04.475875+0000","last_peered":"2023-10-14T01:18:04.475875+0000","last_clean":"2023-10-14T01:18:04.475875+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:18:04.475875+0000","last_undegraded":"2023-10-14T01:18:04.475875+0000","last_fullsized":"2023-10-14T01:18:04.475875+0000","mapping_epoch":13,"log_start":"145'295811","ondisk_log_start":"145'295811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3639185657,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":304824,"num_read_kb":1176488,"num_write":294672,"num_write_kb":1178414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"149'300221","reported_seq":593735,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:03.747088+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:18:03.747088+0000","last_peered":"2023-10-14T01:18:03.747088+0000","last_clean":"2023-10-14T01:18:03.747088+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:18:03.747088+0000","last_undegraded":"2023-10-14T01:18:03.747088+0000","last_fullsized":"2023-10-14T01:18:03.747088+0000","mapping_epoch":13,"log_start":"145'290209","ondisk_log_start":"145'290209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1111,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 462s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3574923448,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":294028,"num_read_kb":1149935,"num_write":290103,"num_write_kb":1160299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":360,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-14T01:17:58.613893+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:17:58.613893+0000","last_peered":"2023-10-14T01:17:58.613893+0000","last_clean":"2023-10-14T01:17:58.613893+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:17:58.613893+0000","last_undegraded":"2023-10-14T01:17:58.613893+0000","last_fullsized":"2023-10-14T01:17:58.613893+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28778453304,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2376828,"num_read_kb":9257330,"num_write":2335144,"num_write_kb":9339105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19125460992,"data_stored":23436939888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80326,"ondisk_log_size":80326,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739350,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4650500,"kb_used_data":3508284,"kb_used_omap":317,"kb_used_meta":1141890,"kb_avail":89721340,"statfs":{"total":96636764160,"available":91874652160,"internally_reserved":0,"allocated":3592482816,"data_stored":4401609582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1169296022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:21 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.98599999999999999,"15min":0.82799999999999996},"min":{"1min":0.27300000000000002,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":5.1529999999999996,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.214},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.77000000000000002,"15min":0.82299999999999995},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.476,"5min":7.6680000000000001,"15min":7.782},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:17:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3859999999999999,"15min":1.3540000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.5199999999999996,"5min":10.726000000000001,"15min":14.103999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.6870000000000001,"15min":1.446},"min":{"1min":0.27100000000000002,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":8.8650000000000002,"5min":9.1920000000000002,"15min":14.586},"last":0.39400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739336,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11004184,"kb_used_data":9340496,"kb_used_omap":757,"kb_used_meta":1662922,"kb_avail":83367656,"statfs":{"total":96636764160,"available":85368479744,"internally_reserved":0,"allocated":9564667904,"data_stored":11720094728,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":775804,"internal_metadata":1702832516},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,6,6,8,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.0349999999999999,"15min":0.749},"min":{"1min":0.23999999999999999,"5min":0.20100000000000001,"15min":0.19400000000000001},"max":{"1min":6.8739999999999997,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.3759999999999999,"15min":0.90600000000000003},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.915,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.51000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:17:57 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.2,"15min":1.165},"min":{"1min":0.24299999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":3.9329999999999998,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.306,"5min":1.403,"15min":1.3680000000000001},"min":{"1min":0.20599999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.1630000000000003,"5min":24.831,"15min":39.412999999999997},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739348,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6993676,"kb_used_data":5831868,"kb_used_omap":425,"kb_used_meta":1161366,"kb_avail":87378164,"statfs":{"total":96636764160,"available":89475239936,"internally_reserved":0,"allocated":5971832832,"data_stored":7317287242,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1189238854},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:43 2023","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.177,"15min":0.95399999999999996},"min":{"1min":0.217,"5min":0.186,"15min":0.17599999999999999},"max":{"1min":11.734999999999999,"5min":11.734999999999999,"15min":11.734999999999999},"last":3.5840000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.68300000000000005,"15min":0.61599999999999999},"min":{"1min":0.23300000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":3.2530000000000001,"5min":5.351,"15min":9.2439999999999998},"last":0.21199999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:17:43 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.0229999999999999,"15min":1.1599999999999999},"min":{"1min":0.23100000000000001,"5min":0.20200000000000001,"15min":0.19800000000000001},"max":{"1min":4.7069999999999999,"5min":9.5809999999999995,"15min":10.539999999999999},"last":3.5419999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.70099999999999996,"15min":0.77300000000000002},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":3.7120000000000002,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.23999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5970968576,"data_stored":7316735610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9563312128,"data_stored":11719051576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3591180288,"data_stored":4401152702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:18:06.830 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:06.830 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:06.831 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:06.831 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:06.832 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:06.832 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:08.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:08.898+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:08.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:08.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:18:08.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:08.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:18:08.917 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:08.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:08.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:08.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:08.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:08.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:08.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:08.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:08.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:08.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:08.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:08.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:08.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:08.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:08.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:08.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:10.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:10.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.486+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:10.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:10.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:10.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:10.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:10.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:10.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:10.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:10.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:10.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:10.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:10.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:11.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:11.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:11.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:11.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:11.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:11.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:11.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:11.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:11.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:11.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:11.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:11.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:11.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:11.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:12.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:12.582+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:12.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:12.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:12.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:12.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:12.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:12.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:12.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:12.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:12.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:12.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:12.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:12.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:12.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:12.914+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:13.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:12.998+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:13.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:13.046+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:14.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:14.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:14.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:14.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:14.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:14.434+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:14.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:14.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:14.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:14.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:14.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:14.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:15.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.290+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:15.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:15.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:15.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:15.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:15.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.590+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:15.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:15.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:15.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:15.650+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:15.740 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:15.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:15.771 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:15.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:15.787 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:15.782+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:15.788 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:15.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:15.788 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:15.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:15.789 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:15.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:16.870 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:16.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:16.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:16.890+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:16.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:16.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:16.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:16.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:16.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:16.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:17.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:17.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:17.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:17.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:17.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:17.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:17.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:17.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:17.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:17.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:19.054 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:19.050+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:18:19.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:19.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:19.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:19.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:19.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:19.074+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:19.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:19.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:19.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:19.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:19.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:19.094+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:20.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:20.662+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:20.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:20.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:20.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:20.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:20.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:20.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:23.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:23.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:23.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:23.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:23.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:23.042+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:23.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:23.058+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:23.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:23.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:23.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:23.062+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:23.788 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:18:23.778+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:18:25.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:25.198+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:25.252 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:25.250+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:25.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:25.306+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:25.322 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:25.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:25.322 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:25.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:25.323 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:25.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:25.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:25.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:18:25.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:25.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:25.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:25.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:25.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:25.979 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:25.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:26.833 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:27.712 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:18:27.713 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:18:27.774 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2615,"stamp":"2023-10-14T01:18:25.763085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28812090200,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2385034,"num_read_kb":9289647,"num_write":2343413,"num_write_kb":9374053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80521,"ondisk_log_size":80521,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22919436,"kb_used_data":18744436,"kb_used_omap":1500,"kb_used_meta":4173475,"kb_avail":260196084,"statfs":{"total":289910292480,"available":266440790016,"internally_reserved":0,"allocated":19194302464,"data_stored":23504310464,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1536928,"internal_metadata":4273638496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,2,6,19,12,25],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23588864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5706,"num_read_kb":22744,"num_write":5779,"num_write_kb":23116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004507"},"pg_stats":[{"pgid":"2.7","version":"149'314442","reported_seq":617394,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:24.477306+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:18:24.477306+0000","last_peered":"2023-10-14T01:18:24.477306+0000","last_clean":"2023-10-14T01:18:24.477306+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:18:24.477306+0000","last_undegraded":"2023-10-14T01:18:24.477306+0000","last_fullsized":"2023-10-14T01:18:24.477306+0000","mapping_epoch":13,"log_start":"145'304402","ondisk_log_start":"145'304402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3644731830,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":304028,"num_read_kb":1178599,"num_write":296121,"num_write_kb":1184242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'314526","reported_seq":618035,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:24.441847+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:18:24.441847+0000","last_peered":"2023-10-14T01:18:24.441847+0000","last_clean":"2023-10-14T01:18:24.441847+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:18:24.441847+0000","last_undegraded":"2023-10-14T01:18:24.441847+0000","last_fullsized":"2023-10-14T01:18:24.441847+0000","mapping_epoch":13,"log_start":"145'304512","ondisk_log_start":"145'304512","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":917,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 476s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3635990896,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":306002,"num_read_kb":1176525,"num_write":295747,"num_write_kb":1182615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'284293","reported_seq":566177,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:24.457557+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:18:24.457557+0000","last_peered":"2023-10-14T01:18:24.457557+0000","last_clean":"2023-10-14T01:18:24.457557+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:18:24.457557+0000","last_undegraded":"2023-10-14T01:18:24.457557+0000","last_fullsized":"2023-10-14T01:18:24.457557+0000","mapping_epoch":13,"log_start":"145'274205","ondisk_log_start":"145'274205","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3480203666,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":282300,"num_read_kb":1113280,"num_write":283675,"num_write_kb":1134670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'277974","reported_seq":556312,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:24.289699+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:18:24.289699+0000","last_peered":"2023-10-14T01:18:24.289699+0000","last_clean":"2023-10-14T01:18:24.289699+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:18:24.289699+0000","last_undegraded":"2023-10-14T01:18:24.289699+0000","last_fullsized":"2023-10-14T01:18:24.289699+0000","mapping_epoch":13,"log_start":"145'267907","ondisk_log_start":"145'267907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":970,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 355s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3404923011,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":278631,"num_read_kb":1096849,"num_write":276840,"num_write_kb":1107282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"149'313428","reported_seq":627681,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:22.398336+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:18:22.398336+0000","last_peered":"2023-10-14T01:18:22.398336+0000","last_clean":"2023-10-14T01:18:22.398336+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:18:22.398336+0000","last_undegraded":"2023-10-14T01:18:22.398336+0000","last_fullsized":"2023-10-14T01:18:22.398336+0000","mapping_epoch":13,"log_start":"145'303409","ondisk_log_start":"145'303409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3829510463,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":314632,"num_read_kb":1232698,"num_write":310982,"num_write_kb":1243781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"149'302337","reported_seq":599926,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:24.564000+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:18:24.564000+0000","last_peered":"2023-10-14T01:18:24.564000+0000","last_clean":"2023-10-14T01:18:24.564000+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:18:24.564000+0000","last_undegraded":"2023-10-14T01:18:24.564000+0000","last_fullsized":"2023-10-14T01:18:24.564000+0000","mapping_epoch":13,"log_start":"145'292309","ondisk_log_start":"145'292309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3593777421,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":298392,"num_read_kb":1156920,"num_write":293082,"num_write_kb":1172114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"149'306880","reported_seq":611803,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:24.559864+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:18:24.559864+0000","last_peered":"2023-10-14T01:18:24.559864+0000","last_clean":"2023-10-14T01:18:24.559864+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:18:24.559864+0000","last_undegraded":"2023-10-14T01:18:24.559864+0000","last_fullsized":"2023-10-14T01:18:24.559864+0000","mapping_epoch":13,"log_start":"145'296811","ondisk_log_start":"145'296811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3643314425,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":305855,"num_read_kb":1180580,"num_write":295682,"num_write_kb":1182454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"149'301278","reported_seq":595843,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:24.294418+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:18:24.294418+0000","last_peered":"2023-10-14T01:18:24.294418+0000","last_clean":"2023-10-14T01:18:24.294418+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:18:24.294418+0000","last_undegraded":"2023-10-14T01:18:24.294418+0000","last_fullsized":"2023-10-14T01:18:24.294418+0000","mapping_epoch":13,"log_start":"145'291209","ondisk_log_start":"145'291209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1145,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 483s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3579048120,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":295068,"num_read_kb":1154087,"num_write":291115,"num_write_kb":1164347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":360,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-14T01:17:58.613893+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:17:58.613893+0000","last_peered":"2023-10-14T01:17:58.613893+0000","last_clean":"2023-10-14T01:17:58.613893+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:17:58.613893+0000","last_undegraded":"2023-10-14T01:17:58.613893+0000","last_fullsized":"2023-10-14T01:17:58.613893+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28811499832,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2384908,"num_read_kb":9289538,"num_write":2343244,"num_write_kb":9371505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19191681024,"data_stored":23503159920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80394,"ondisk_log_size":80394,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739354,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4676272,"kb_used_data":3520168,"kb_used_omap":317,"kb_used_meta":1155778,"kb_avail":89695568,"statfs":{"total":96636764160,"available":91848261632,"internally_reserved":0,"allocated":3604652032,"data_stored":4413778798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1183517334},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:21 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.98599999999999999,"15min":0.82799999999999996},"min":{"1min":0.27300000000000002,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":5.1529999999999996,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.77000000000000002,"15min":0.82299999999999995},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.476,"5min":7.6680000000000001,"15min":7.782},"last":0.26300000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:17:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.3859999999999999,"15min":1.3540000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.5199999999999996,"5min":10.726000000000001,"15min":14.103999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.6870000000000001,"15min":1.446},"min":{"1min":0.27100000000000002,"5min":0.22800000000000001,"15min":0.20699999999999999},"max":{"1min":8.8650000000000002,"5min":9.1920000000000002,"15min":14.586},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739340,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11166564,"kb_used_data":9372380,"kb_used_omap":757,"kb_used_meta":1793418,"kb_avail":83205276,"statfs":{"total":96636764160,"available":85202202624,"internally_reserved":0,"allocated":9597317120,"data_stored":11752743944,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":775804,"internal_metadata":1836460420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,1,6,12,6,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:38 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":1.0349999999999999,"15min":0.749},"min":{"1min":0.23999999999999999,"5min":0.20100000000000001,"15min":0.19400000000000001},"max":{"1min":6.8739999999999997,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.438},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.3759999999999999,"15min":0.90600000000000003},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":6.915,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.47199999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:17:57 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.2,"15min":1.165},"min":{"1min":0.24299999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":3.9329999999999998,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.505},{"interface":"front","average":{"1min":1.306,"5min":1.403,"15min":1.3680000000000001},"min":{"1min":0.20599999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.1630000000000003,"5min":24.831,"15min":39.412999999999997},"last":0.255}]}]},{"osd":0,"up_from":8,"seq":34359739352,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7076600,"kb_used_data":5851888,"kb_used_omap":425,"kb_used_meta":1224278,"kb_avail":87295240,"statfs":{"total":96636764160,"available":89390325760,"internally_reserved":0,"allocated":5992333312,"data_stored":7337787722,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1253660742},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,5,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:43 2023","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.177,"15min":0.95399999999999996},"min":{"1min":0.217,"5min":0.186,"15min":0.17599999999999999},"max":{"1min":11.734999999999999,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.627},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.68300000000000005,"15min":0.61599999999999999},"min":{"1min":0.23300000000000001,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":3.2530000000000001,"5min":5.351,"15min":9.2439999999999998},"last":0.311}]},{"osd":2,"last update":"Sat Oct 14 01:17:43 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.0229999999999999,"15min":1.1599999999999999},"min":{"1min":0.23100000000000001,"5min":0.20200000000000001,"15min":0.19800000000000001},"max":{"1min":4.7069999999999999,"5min":9.5809999999999995,"15min":10.539999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.70099999999999996,"15min":0.77300000000000002},"min":{"1min":0.28100000000000003,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":3.7120000000000002,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5991419904,"data_stored":7337186938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9596866560,"data_stored":11752606008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3603394560,"data_stored":4413366974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:18:27.775 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:27.776 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:27.776 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:27.777 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:27.777 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:27.777 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:29.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:29.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:29.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:29.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:29.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:29.194+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:30.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:30.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:31.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:31.026+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:31.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:31.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:31.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:31.102+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:31.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:31.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:31.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:31.122+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:31.155 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:31.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:31.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:31.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:31.156 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:31.150+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:32.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:32.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:32.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:32.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:32.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:32.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:32.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:32.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:32.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:32.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:32.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:33.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:33.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:18:33.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:33.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:33.959 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:33.954+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:34.365 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:34.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:18:34.430 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:34.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:34.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:34.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:34.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:34.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:34.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:34.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:34.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:34.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:34.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:34.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:35.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:35.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:35.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:35.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:35.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:35.866+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:35.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:35.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:35.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:35.914+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:36.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:36.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:18:36.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:36.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:36.791 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:36.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:36.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:36.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:36.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:36.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:36.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:36.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:36.830 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:36.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:37.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:37.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:37.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:37.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:37.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:37.994+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:38.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:38.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:38.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:38.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:38.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:38.154+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:38.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:38.182+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:38.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:38.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:38.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:38.614+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:38.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:38.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:40.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:40.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:18:40.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:40.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:40.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:40.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:40.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:40.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:40.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:40.518+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:41.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:41.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:41.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:41.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:41.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:41.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:41.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:41.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:41.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:41.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:41.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:41.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:42.374 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:42.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:18:42.405 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:42.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:42.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:42.418+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:42.452 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:42.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:42.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:42.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:42.453 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:42.446+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:42.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:42.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:42.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:42.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:45.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:45.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:45.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:45.449+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:45.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:45.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:45.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:45.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:47.154 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:47.149+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:47.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:47.161+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:47.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:47.169+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:47.191 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:47.185+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:47.192 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:47.185+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:47.778 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:48.469 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:18:48.471 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:18:48.566 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2626,"stamp":"2023-10-14T01:18:47.770011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28846394200,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2393264,"num_read_kb":9322483,"num_write":2351817,"num_write_kb":9407669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80526,"ondisk_log_size":80526,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23269604,"kb_used_data":18815052,"kb_used_omap":1500,"kb_used_meta":4453027,"kb_avail":259845916,"statfs":{"total":289910292480,"available":266082217984,"internally_reserved":0,"allocated":19266613248,"data_stored":23576621248,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1536928,"internal_metadata":4559899744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,6,14,9,17,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17674240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4242,"num_read_kb":16928,"num_write":4335,"num_write_kb":17340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005182"},"pg_stats":[{"pgid":"2.7","version":"149'315534","reported_seq":619564,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:45.100126+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:18:45.100126+0000","last_peered":"2023-10-14T01:18:45.100126+0000","last_clean":"2023-10-14T01:18:45.100126+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:18:45.100126+0000","last_undegraded":"2023-10-14T01:18:45.100126+0000","last_fullsized":"2023-10-14T01:18:45.100126+0000","mapping_epoch":13,"log_start":"146'305502","ondisk_log_start":"146'305502","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3648979382,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":305090,"num_read_kb":1182835,"num_write":297161,"num_write_kb":1188402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"149'315665","reported_seq":620237,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:45.113413+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:18:45.113413+0000","last_peered":"2023-10-14T01:18:45.113413+0000","last_clean":"2023-10-14T01:18:45.113413+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:18:45.113413+0000","last_undegraded":"2023-10-14T01:18:45.113413+0000","last_fullsized":"2023-10-14T01:18:45.113413+0000","mapping_epoch":13,"log_start":"146'305612","ondisk_log_start":"146'305612","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":956,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 497s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3640324464,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":307044,"num_read_kb":1180681,"num_write":296813,"num_write_kb":1186879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"149'285289","reported_seq":568120,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:45.112266+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:18:45.112266+0000","last_peered":"2023-10-14T01:18:45.112266+0000","last_clean":"2023-10-14T01:18:45.112266+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:18:45.112266+0000","last_undegraded":"2023-10-14T01:18:45.112266+0000","last_fullsized":"2023-10-14T01:18:45.112266+0000","mapping_epoch":13,"log_start":"145'275205","ondisk_log_start":"145'275205","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3484266898,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":283247,"num_read_kb":1117056,"num_write":284671,"num_write_kb":1138654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"149'278993","reported_seq":558304,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:45.077801+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:18:45.077801+0000","last_peered":"2023-10-14T01:18:45.077801+0000","last_clean":"2023-10-14T01:18:45.077801+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:18:45.077801+0000","last_undegraded":"2023-10-14T01:18:45.077801+0000","last_fullsized":"2023-10-14T01:18:45.077801+0000","mapping_epoch":13,"log_start":"145'268907","ondisk_log_start":"145'268907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1038,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 375s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3408896131,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":279599,"num_read_kb":1100709,"num_write":277813,"num_write_kb":1111174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"149'314731","reported_seq":630226,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:47.507872+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:18:47.507872+0000","last_peered":"2023-10-14T01:18:47.507872+0000","last_clean":"2023-10-14T01:18:47.507872+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:18:47.507872+0000","last_undegraded":"2023-10-14T01:18:47.507872+0000","last_fullsized":"2023-10-14T01:18:47.507872+0000","mapping_epoch":13,"log_start":"146'304709","ondisk_log_start":"146'304709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3834806591,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":315869,"num_read_kb":1237638,"num_write":312282,"num_write_kb":1248981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"149'303334","reported_seq":601908,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:44.736418+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:18:44.736418+0000","last_peered":"2023-10-14T01:18:44.736418+0000","last_clean":"2023-10-14T01:18:44.736418+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:18:44.736418+0000","last_undegraded":"2023-10-14T01:18:44.736418+0000","last_fullsized":"2023-10-14T01:18:44.736418+0000","mapping_epoch":13,"log_start":"145'293309","ondisk_log_start":"145'293309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3597689101,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":299365,"num_read_kb":1160804,"num_write":294047,"num_write_kb":1175974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"149'307962","reported_seq":613901,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:18:44.727383+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:18:44.727383+0000","last_peered":"2023-10-14T01:18:44.727383+0000","last_clean":"2023-10-14T01:18:44.727383+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:18:44.727383+0000","last_undegraded":"2023-10-14T01:18:44.727383+0000","last_fullsized":"2023-10-14T01:18:44.727383+0000","mapping_epoch":13,"log_start":"145'297911","ondisk_log_start":"145'297911","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3647607033,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":306847,"num_read_kb":1184532,"num_write":296724,"num_write_kb":1186622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"149'302355","reported_seq":597948,"reported_epoch":149,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:45.030854+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:18:45.030854+0000","last_peered":"2023-10-14T01:18:45.030854+0000","last_clean":"2023-10-14T01:18:45.030854+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:18:45.030854+0000","last_undegraded":"2023-10-14T01:18:45.030854+0000","last_fullsized":"2023-10-14T01:18:45.030854+0000","mapping_epoch":13,"log_start":"145'292309","ondisk_log_start":"145'292309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1200,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 503s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3583234232,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":296077,"num_read_kb":1158119,"num_write":292137,"num_write_kb":1168435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":360,"reported_epoch":148,"state":"active+clean","last_fresh":"2023-10-14T01:17:58.613893+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:17:58.613893+0000","last_peered":"2023-10-14T01:17:58.613893+0000","last_clean":"2023-10-14T01:17:58.613893+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:17:58.613893+0000","last_undegraded":"2023-10-14T01:17:58.613893+0000","last_fullsized":"2023-10-14T01:17:58.613893+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28845803832,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2393138,"num_read_kb":9322374,"num_write":2351648,"num_write_kb":9405121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19263131648,"data_stored":23574610544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80399,"ondisk_log_size":80399,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739358,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4738140,"kb_used_data":3532052,"kb_used_omap":317,"kb_used_meta":1205762,"kb_avail":89633700,"statfs":{"total":96636764160,"available":91784908800,"internally_reserved":0,"allocated":3616821248,"data_stored":4425948014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1234700950},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:26 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.86099999999999999,"15min":0.80100000000000005},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":1.919,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.79500000000000004,"15min":0.78900000000000003},"min":{"1min":0.247,"5min":0.19900000000000001,"15min":0.19800000000000001},"max":{"1min":2.1190000000000002,"5min":7.6680000000000001,"15min":7.782},"last":0.33400000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:18:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.329,"15min":1.3120000000000001},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.4889999999999999,"5min":10.726000000000001,"15min":11.077999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.5089999999999999,"15min":1.4319999999999999},"min":{"1min":0.23100000000000001,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":3.1349999999999998,"5min":8.8650000000000002,"15min":14.586},"last":5.7089999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739344,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11329728,"kb_used_data":9405944,"kb_used_omap":757,"kb_used_meta":1923018,"kb_avail":83042112,"statfs":{"total":96636764160,"available":85035122688,"internally_reserved":0,"allocated":9631686656,"data_stored":11787113480,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":775804,"internal_metadata":1969170820},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,8,4,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.1100000000000001,"15min":0.78600000000000003},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":10.02,"5min":23.434000000000001,"15min":23.434000000000001},"last":4.0140000000000002},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.52,"15min":0.95799999999999996},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":10.074999999999999,"5min":24.908000000000001,"15min":24.908000000000001},"last":3.98}]},{"osd":2,"last update":"Sat Oct 14 01:17:57 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.2,"15min":1.165},"min":{"1min":0.24299999999999999,"5min":0.23499999999999999,"15min":0.222},"max":{"1min":3.9329999999999998,"5min":24.870000000000001,"15min":43.871000000000002},"last":6.8570000000000002},{"interface":"front","average":{"1min":1.306,"5min":1.403,"15min":1.3680000000000001},"min":{"1min":0.20599999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.1630000000000003,"5min":24.831,"15min":39.412999999999997},"last":4.484}]}]},{"osd":0,"up_from":8,"seq":34359739357,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7201736,"kb_used_data":5877056,"kb_used_omap":425,"kb_used_meta":1324246,"kb_avail":87170104,"statfs":{"total":96636764160,"available":89262186496,"internally_reserved":0,"allocated":6018105344,"data_stored":7363559754,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1356027974},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.1060000000000001,"15min":0.90900000000000003},"min":{"1min":0.23000000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":3.5840000000000001,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.505,"5min":0.624,"15min":0.58499999999999996},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":3.0270000000000001,"5min":5.351,"15min":9.2439999999999998},"last":0.20999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:18:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.0940000000000001,"15min":1.1579999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19800000000000001},"max":{"1min":9.8059999999999992,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.308},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.624,"15min":0.75600000000000001},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":3.7770000000000001,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6016925696,"data_stored":7362692730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9630515200,"data_stored":11786254648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3615690752,"data_stored":4425663166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:18:48.568 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:48.569 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:48.570 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:48.570 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:48.571 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:48.571 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:48.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:48.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:48.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:48.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:48.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:48.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:48.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:48.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:48.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:48.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:48.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:48.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:48.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:48.945+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:50.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:50.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:50.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:50.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:50.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:50.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:50.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:50.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:50.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:50.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:50.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:50.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:50.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:50.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:52.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:52.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:52.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:52.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:52.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:52.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:54.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:54.277+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:54.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:54.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:55.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:55.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:55.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:55.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:55.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:18:55.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:18:55.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:55.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:55.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:55.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:55.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:55.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:55.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:56.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:56.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:18:56.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:56.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:18:56.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:56.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:56.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:56.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:56.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:56.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:56.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:56.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:57.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:57.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:18:57.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:57.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:59.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:59.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:59.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:59.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:18:59.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:59.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:18:59.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:59.349+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:18:59.374 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:59.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:18:59.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:59.377+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:18:59.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:59.377+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:18:59.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:18:59.377+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:18:59.402 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:59.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:18:59.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:18:59.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:00.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:00.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:00.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:00.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:00.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:00.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:01.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:01.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:01.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:01.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:01.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:01.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:01.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:01.357+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:01.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:01.369+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:01.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:01.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:01.388 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:01.385+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:01.388 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:01.385+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:01.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:01.385+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:01.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:01.385+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:03.403 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:03.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:03.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:03.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:04.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:04.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:04.375 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:04.365+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:04.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:04.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:04.396 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:04.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:04.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:04.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:04.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:04.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:04.501 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:04.497+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:04.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:04.497+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:04.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:04.497+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:04.994 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:04.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:05.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:05.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:05.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:05.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:05.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:05.029+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:06.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:06.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:06.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:06.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:06.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:06.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:06.798 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:06.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:06.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:06.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:06.815 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:06.809+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:06.815 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:06.809+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:06.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:06.825+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:06.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:06.825+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:06.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:06.825+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:07.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.081+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:19:07.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:07.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:07.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:07.255 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.249+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:07.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:07.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:07.265 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:07.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:07.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:07.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:07.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:07.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:07.524 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:07.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:07.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.721+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:07.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:07.779 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:07.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:19:07.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:07.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:08.573 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:19:08.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:19:08.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:19:08.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:08.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:08.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:08.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:08.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:08.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:08.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:08.965+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:09.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:09.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:09.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:09.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:09.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:09.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:09.438 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:19:09.439 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:19:09.482 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2637,"stamp":"2023-10-14T01:19:07.773340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28879207256,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2401148,"num_read_kb":9353975,"num_write":2359861,"num_write_kb":9439845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80470,"ondisk_log_size":80470,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23519096,"kb_used_data":18877728,"kb_used_omap":1500,"kb_used_meta":4639843,"kb_avail":259596424,"statfs":{"total":289910292480,"available":265826738176,"internally_reserved":0,"allocated":19330793472,"data_stored":23640789388,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1536928,"internal_metadata":4751199328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,12,5,13,34,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16478208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3977,"num_read_kb":15888,"num_write":4045,"num_write_kb":16180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001633"},"pg_stats":[{"pgid":"2.7","version":"150'316643","reported_seq":621671,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:05.692522+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:19:05.692522+0000","last_peered":"2023-10-14T01:19:05.692522+0000","last_clean":"2023-10-14T01:19:05.692522+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:19:05.692522+0000","last_undegraded":"2023-10-14T01:19:05.692522+0000","last_fullsized":"2023-10-14T01:19:05.692522+0000","mapping_epoch":13,"log_start":"146'306602","ondisk_log_start":"146'306602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3653190070,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":306073,"num_read_kb":1186763,"num_write":298194,"num_write_kb":1192534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'316831","reported_seq":622405,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:05.693574+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:19:05.693574+0000","last_peered":"2023-10-14T01:19:05.693574+0000","last_clean":"2023-10-14T01:19:05.693574+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:19:05.693574+0000","last_undegraded":"2023-10-14T01:19:05.693574+0000","last_fullsized":"2023-10-14T01:19:05.693574+0000","mapping_epoch":13,"log_start":"146'306812","ondisk_log_start":"146'306812","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":984,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 517s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3644682608,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":308020,"num_read_kb":1184585,"num_write":297887,"num_write_kb":1191175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'286288","reported_seq":570104,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:05.717002+0000","last_change":"2023-10-13T23:57:08.679755+0000","last_active":"2023-10-14T01:19:05.717002+0000","last_peered":"2023-10-14T01:19:05.717002+0000","last_clean":"2023-10-14T01:19:05.717002+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:19:05.717002+0000","last_undegraded":"2023-10-14T01:19:05.717002+0000","last_fullsized":"2023-10-14T01:19:05.717002+0000","mapping_epoch":13,"log_start":"146'276205","ondisk_log_start":"146'276205","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3488321938,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":284233,"num_read_kb":1120996,"num_write":285670,"num_write_kb":1142650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'279983","reported_seq":560262,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:05.696963+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:19:05.696963+0000","last_peered":"2023-10-14T01:19:05.696963+0000","last_clean":"2023-10-14T01:19:05.696963+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:19:05.696963+0000","last_undegraded":"2023-10-14T01:19:05.696963+0000","last_fullsized":"2023-10-14T01:19:05.696963+0000","mapping_epoch":13,"log_start":"146'269907","ondisk_log_start":"146'269907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1090,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 396s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3412828291,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":280552,"num_read_kb":1104513,"num_write":278774,"num_write_kb":1115018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"150'315755","reported_seq":632285,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:07.740833+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:19:07.740833+0000","last_peered":"2023-10-14T01:19:07.740833+0000","last_clean":"2023-10-14T01:19:07.740833+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:19:07.740833+0000","last_undegraded":"2023-10-14T01:19:07.740833+0000","last_fullsized":"2023-10-14T01:19:07.740833+0000","mapping_epoch":13,"log_start":"146'305709","ondisk_log_start":"146'305709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3838996799,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":316899,"num_read_kb":1241750,"num_write":313303,"num_write_kb":1253065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"150'304355","reported_seq":603966,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:04.870490+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:19:04.870490+0000","last_peered":"2023-10-14T01:19:04.870490+0000","last_clean":"2023-10-14T01:19:04.870490+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:19:04.870490+0000","last_undegraded":"2023-10-14T01:19:04.870490+0000","last_fullsized":"2023-10-14T01:19:04.870490+0000","mapping_epoch":13,"log_start":"146'294309","ondisk_log_start":"146'294309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3601764621,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":300389,"num_read_kb":1164888,"num_write":295052,"num_write_kb":1179994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"150'309015","reported_seq":615924,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:04.869670+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:19:04.869670+0000","last_peered":"2023-10-14T01:19:04.869670+0000","last_clean":"2023-10-14T01:19:04.869670+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:19:04.869670+0000","last_undegraded":"2023-10-14T01:19:04.869670+0000","last_fullsized":"2023-10-14T01:19:04.869670+0000","mapping_epoch":13,"log_start":"146'299011","ondisk_log_start":"146'299011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3651834105,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":307800,"num_read_kb":1188344,"num_write":297757,"num_write_kb":1190754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"150'303337","reported_seq":599929,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:05.652388+0000","last_change":"2023-10-14T01:10:21.109901+0000","last_active":"2023-10-14T01:19:05.652388+0000","last_peered":"2023-10-14T01:19:05.652388+0000","last_clean":"2023-10-14T01:19:05.652388+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:19:05.652388+0000","last_undegraded":"2023-10-14T01:19:05.652388+0000","last_fullsized":"2023-10-14T01:19:05.652388+0000","mapping_epoch":13,"log_start":"146'293309","ondisk_log_start":"146'293309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1249,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 524s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3586998456,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":297056,"num_read_kb":1162027,"num_write":293055,"num_write_kb":1172107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":361,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:17:59.733796+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:17:59.733796+0000","last_peered":"2023-10-14T01:17:59.733796+0000","last_clean":"2023-10-14T01:17:59.733796+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:17:59.733796+0000","last_undegraded":"2023-10-14T01:17:59.733796+0000","last_fullsized":"2023-10-14T01:17:59.733796+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28878616888,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2401022,"num_read_kb":9353866,"num_write":2359692,"num_write_kb":9437297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":6440,"num_omap_keys":120,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19328811008,"data_stored":23640289904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80343,"ondisk_log_size":80343,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739362,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4763600,"kb_used_data":3543624,"kb_used_omap":317,"kb_used_meta":1219650,"kb_avail":89608240,"statfs":{"total":96636764160,"available":91758837760,"internally_reserved":0,"allocated":3628670976,"data_stored":4437793714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1248922262},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:26 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.86099999999999999,"15min":0.80100000000000005},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":1.919,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.79500000000000004,"15min":0.78900000000000003},"min":{"1min":0.247,"5min":0.19900000000000001,"15min":0.19800000000000001},"max":{"1min":2.1190000000000002,"5min":7.6680000000000001,"15min":7.782},"last":0.312}]},{"osd":1,"last update":"Sat Oct 14 01:18:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.329,"15min":1.3120000000000001},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.4889999999999999,"5min":10.726000000000001,"15min":11.077999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.5089999999999999,"15min":1.4319999999999999},"min":{"1min":0.23100000000000001,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":3.1349999999999998,"5min":8.8650000000000002,"15min":14.586},"last":0.27300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739348,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11467112,"kb_used_data":9437472,"kb_used_omap":757,"kb_used_meta":2028874,"kb_avail":82904728,"statfs":{"total":96636764160,"available":84894441472,"internally_reserved":0,"allocated":9663971328,"data_stored":11819394124,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":775804,"internal_metadata":2077567364},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,3,10,28,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.1100000000000001,"15min":0.78600000000000003},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":10.02,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.52,"15min":0.95799999999999996},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":10.074999999999999,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3049999999999999,"15min":1.1970000000000001},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":10.112,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.143,"15min":1.349},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.484,"5min":24.831,"15min":39.412999999999997},"last":3.6000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739361,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7288384,"kb_used_data":5896632,"kb_used_omap":425,"kb_used_meta":1391318,"kb_avail":87083456,"statfs":{"total":96636764160,"available":89173458944,"internally_reserved":0,"allocated":6038151168,"data_stored":7383601550,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1424709702},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.1060000000000001,"15min":0.90900000000000003},"min":{"1min":0.23000000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":3.5840000000000001,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.248},{"interface":"front","average":{"1min":0.505,"5min":0.624,"15min":0.58499999999999996},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":3.0270000000000001,"5min":5.351,"15min":9.2439999999999998},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:18:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.0940000000000001,"15min":1.1579999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19800000000000001},"max":{"1min":9.8059999999999992,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.624,"15min":0.75600000000000001},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":3.7770000000000001,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6037401600,"data_stored":7383168634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9663692800,"data_stored":11819432248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3627716608,"data_stored":4437689022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:19:09.483 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:09.484 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:09.484 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:09.484 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:09.485 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:09.485 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:19:10.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:10.233+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:10.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:10.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:10.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:10.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:10.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:10.249+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:10.285 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:19:10.337 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-14T01:19:10.459 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:10.457+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:10.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:10.501+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:10.516 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:10.517+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:10.523 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:10.517+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:10.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:10.517+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:10.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:10.517+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:10.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:10.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:19:10.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:10.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:19:10.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:10.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:19:11.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:11.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:11.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:11.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:11.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:11.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:11.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:11.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:11.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:11.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:11.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:11.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:11.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:11.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:11.314 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-14T01:19:11.351 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:19:12.066 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:19:12.121 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-14T01:19:12.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:19:12.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:19:12.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:12.165 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:12.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:12.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:12.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:12.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:12.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:12.165+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:12.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:12.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:12.908 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-14T01:19:12.947 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-14T01:19:13.185 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:13.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:13.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:13.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:13.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:13.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:13.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:13.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:13.387 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:13.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:13.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:13.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:13.643 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-14T01:19:13.695 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:14.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:14.429+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:14.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:14.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:14.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:14.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:14.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:14.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:15.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:15.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:15.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:15.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:15.301 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:15.297+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:15.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:15.309+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:15.310 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:15.309+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:15.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:15.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:15.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:15.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:15.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:16.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:16.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:16.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:16.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:16.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:16.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:16.917 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:16.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:16.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:16.913+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:18.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:18.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:18.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:18.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:18.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:18.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:19.488 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:19.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:19.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:19.489+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:19.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:19.505+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:19.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:19.505+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:19.508 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:19.505+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:22.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:22.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:22.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:22.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:22.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:22.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:22.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:22.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:23.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:19:23.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:23.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:23.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:23.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:23.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:23.712 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.713+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:23.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:23.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:23.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:23.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:23.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:23.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:23.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:23.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:27.435 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:27.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:27.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:27.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:27.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:27.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:27.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:27.661+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:27.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:27.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:27.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:27.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:27.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:27.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:30.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:30.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:30.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:30.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:30.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.413+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:30.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:30.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:30.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:30.465+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:33.696 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:19:34.545 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:19:34.545 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:19:34.610 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2650,"stamp":"2023-10-14T01:19:33.781650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28919237464,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2410851,"num_read_kb":9392703,"num_write":2369672,"num_write_kb":9479089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7456,"num_omap_keys":140,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80506,"ondisk_log_size":80506,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22941392,"kb_used_data":18956080,"kb_used_omap":1502,"kb_used_meta":3983777,"kb_avail":260174128,"statfs":{"total":289910292480,"available":266418307072,"internally_reserved":0,"allocated":19411025920,"data_stored":23721021836,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":4079388113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,4,3,11,15,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22204416,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5328,"num_read_kb":21268,"num_write":5431,"num_write_kb":21724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005929"},"pg_stats":[{"pgid":"2.7","version":"150'318003","reported_seq":624329,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:32.115132+0000","last_change":"2023-10-13T23:57:08.678930+0000","last_active":"2023-10-14T01:19:32.115132+0000","last_peered":"2023-10-14T01:19:32.115132+0000","last_clean":"2023-10-14T01:19:32.115132+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:19:32.115132+0000","last_undegraded":"2023-10-14T01:19:32.115132+0000","last_fullsized":"2023-10-14T01:19:32.115132+0000","mapping_epoch":13,"log_start":"146'308002","ondisk_log_start":"146'308002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3658371510,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":307347,"num_read_kb":1191851,"num_write":299468,"num_write_kb":1197630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'318257","reported_seq":625105,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:32.057879+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:19:32.057879+0000","last_peered":"2023-10-14T01:19:32.057879+0000","last_clean":"2023-10-14T01:19:32.057879+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:19:32.057879+0000","last_undegraded":"2023-10-14T01:19:32.057879+0000","last_fullsized":"2023-10-14T01:19:32.057879+0000","mapping_epoch":13,"log_start":"146'308213","ondisk_log_start":"146'308213","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1040,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 544s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3650044272,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":309266,"num_read_kb":1189561,"num_write":299199,"num_write_kb":1196423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'287492","reported_seq":572509,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:32.116334+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:19:32.116334+0000","last_peered":"2023-10-14T01:19:32.116334+0000","last_clean":"2023-10-14T01:19:32.116334+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:19:32.116334+0000","last_undegraded":"2023-10-14T01:19:32.116334+0000","last_fullsized":"2023-10-14T01:19:32.116334+0000","mapping_epoch":13,"log_start":"146'277405","ondisk_log_start":"146'277405","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":17,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 10s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3493245330,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":285433,"num_read_kb":1125788,"num_write":286865,"num_write_kb":1147430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'281218","reported_seq":562694,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:32.064930+0000","last_change":"2023-10-14T01:12:29.125580+0000","last_active":"2023-10-14T01:19:32.064930+0000","last_peered":"2023-10-14T01:19:32.064930+0000","last_clean":"2023-10-14T01:19:32.064930+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:19:32.064930+0000","last_undegraded":"2023-10-14T01:19:32.064930+0000","last_fullsized":"2023-10-14T01:19:32.064930+0000","mapping_epoch":13,"log_start":"146'271207","ondisk_log_start":"146'271207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1207,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 422s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3417632899,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":281732,"num_read_kb":1109217,"num_write":279947,"num_write_kb":1119710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"150'317076","reported_seq":634899,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:33.491913+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:19:33.491913+0000","last_peered":"2023-10-14T01:19:33.491913+0000","last_clean":"2023-10-14T01:19:33.491913+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:19:33.491913+0000","last_undegraded":"2023-10-14T01:19:33.491913+0000","last_fullsized":"2023-10-14T01:19:33.491913+0000","mapping_epoch":13,"log_start":"146'307009","ondisk_log_start":"146'307009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3844346175,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":318187,"num_read_kb":1246882,"num_write":314619,"num_write_kb":1258329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"150'305557","reported_seq":606362,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:29.983289+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:19:29.983289+0000","last_peered":"2023-10-14T01:19:29.983289+0000","last_clean":"2023-10-14T01:19:29.983289+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:19:29.983289+0000","last_undegraded":"2023-10-14T01:19:29.983289+0000","last_fullsized":"2023-10-14T01:19:29.983289+0000","mapping_epoch":13,"log_start":"146'295509","ondisk_log_start":"146'295509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3606483213,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":301562,"num_read_kb":1169576,"num_write":296214,"num_write_kb":1184642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"150'310277","reported_seq":618370,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:30.038072+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:19:30.038072+0000","last_peered":"2023-10-14T01:19:30.038072+0000","last_clean":"2023-10-14T01:19:30.038072+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:19:30.038072+0000","last_undegraded":"2023-10-14T01:19:30.038072+0000","last_fullsized":"2023-10-14T01:19:30.038072+0000","mapping_epoch":13,"log_start":"146'300211","ondisk_log_start":"146'300211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3656765689,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":308957,"num_read_kb":1192960,"num_write":298969,"num_write_kb":1195602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"150'304564","reported_seq":602363,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:32.061022+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:19:32.061022+0000","last_peered":"2023-10-14T01:19:32.061022+0000","last_clean":"2023-10-14T01:19:32.061022+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:19:32.061022+0000","last_undegraded":"2023-10-14T01:19:32.061022+0000","last_fullsized":"2023-10-14T01:19:32.061022+0000","mapping_epoch":13,"log_start":"146'294509","ondisk_log_start":"146'294509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3591758008,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":298241,"num_read_kb":1166759,"num_write":294222,"num_write_kb":1176775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":361,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:17:59.733796+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:17:59.733796+0000","last_peered":"2023-10-14T01:17:59.733796+0000","last_clean":"2023-10-14T01:17:59.733796+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:17:59.733796+0000","last_undegraded":"2023-10-14T01:17:59.733796+0000","last_fullsized":"2023-10-14T01:17:59.733796+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28918647096,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2410725,"num_read_kb":9392594,"num_write":2369503,"num_write_kb":9476541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7456,"num_omap_keys":140,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19409113088,"data_stored":23720591984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80379,"ondisk_log_size":80379,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739367,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4868080,"kb_used_data":3557924,"kb_used_omap":317,"kb_used_meta":1309826,"kb_avail":89503760,"statfs":{"total":96636764160,"available":91651850240,"internally_reserved":0,"allocated":3643314176,"data_stored":4452436914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1341262486},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:26 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.86099999999999999,"15min":0.80100000000000005},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16300000000000001},"max":{"1min":1.919,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":1.9399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.79500000000000004,"15min":0.78900000000000003},"min":{"1min":0.247,"5min":0.19900000000000001,"15min":0.19800000000000001},"max":{"1min":2.1190000000000002,"5min":7.6680000000000001,"15min":7.782},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:18:26 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.329,"15min":1.3120000000000001},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.4889999999999999,"5min":10.726000000000001,"15min":11.077999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.5089999999999999,"15min":1.4319999999999999},"min":{"1min":0.23100000000000001,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":3.1349999999999998,"5min":8.8650000000000002,"15min":14.586},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739353,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10647832,"kb_used_data":9476876,"kb_used_omap":759,"kb_used_meta":1170184,"kb_avail":83724008,"statfs":{"total":96636764160,"available":85733384192,"internally_reserved":0,"allocated":9704321024,"data_stored":11859743820,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":1198269173},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,3,1,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:38 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.1100000000000001,"15min":0.78600000000000003},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":10.02,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.52,"15min":0.95799999999999996},"min":{"1min":0.26300000000000001,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":10.074999999999999,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.56899999999999995}]},{"osd":2,"last update":"Sat Oct 14 01:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3049999999999999,"15min":1.1970000000000001},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":10.112,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.143,"15min":1.349},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.484,"5min":24.831,"15min":39.412999999999997},"last":0.45900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739366,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7425480,"kb_used_data":5921280,"kb_used_omap":425,"kb_used_meta":1503766,"kb_avail":86946360,"statfs":{"total":96636764160,"available":89033072640,"internally_reserved":0,"allocated":6063390720,"data_stored":7408841102,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1539856454},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.1060000000000001,"15min":0.90900000000000003},"min":{"1min":0.23000000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":3.5840000000000001,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.505,"5min":0.624,"15min":0.58499999999999996},"min":{"1min":0.20999999999999999,"5min":0.19900000000000001,"15min":0.17399999999999999},"max":{"1min":3.0270000000000001,"5min":5.351,"15min":9.2439999999999998},"last":0.24199999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:18:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.0940000000000001,"15min":1.1579999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19800000000000001},"max":{"1min":9.8059999999999992,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.624,"15min":0.75600000000000001},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.20399999999999999},"max":{"1min":3.7770000000000001,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6062649344,"data_stored":7408416378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9704476672,"data_stored":11860216120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3641987072,"data_stored":4451959486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:19:34.612 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:34.612 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:34.612 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:34.613 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:34.613 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:35.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:19:35.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:35.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:35.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:35.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:35.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.301+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:35.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:35.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:35.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:35.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:35.669 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:35.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:35.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:35.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:35.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:35.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:35.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:36.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:36.005+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:36.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:36.017+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:36.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:36.369+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:36.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:36.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:36.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:36.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:36.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:36.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:40.026 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:40.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:40.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:40.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:40.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:40.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:40.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:40.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:40.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:40.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:40.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:40.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:41.460 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:41.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:41.475 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:41.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:41.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:41.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:41.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:41.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:41.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:41.484+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:41.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:41.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:41.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:41.508+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:43.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:43.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:43.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:43.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:43.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:43.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:43.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:43.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:43.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:43.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:43.397 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:43.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:43.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:43.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:43.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:43.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:43.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:43.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:43.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:43.396+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:46.901 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:46.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:46.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:46.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:46.939 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:46.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:46.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:46.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:46.960 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:46.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:46.961 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:46.960+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:47.341 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.332+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:47.362 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.360+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:47.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:47.381 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:47.464 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:19:47.460+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:19:47.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:47.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.940+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:47.967 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:47.998 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:47.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:48.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:48.008+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:48.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:48.008+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:48.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:48.008+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:49.078 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:49.626 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:49.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:49.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:49.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:49.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:49.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:49.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:49.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:49.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:49.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:49.974 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:49.972+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:49.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:49.984+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:49.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:49.984+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:49.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:49.984+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:51.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:51.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:51.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:51.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:51.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:51.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:51.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:51.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:51.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:51.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:51.207 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:51.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:51.814 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:51.812+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:51.833 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:51.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:51.850 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:51.848+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:51.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:51.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:51.863 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:51.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:51.864 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:51.864+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:52.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:52.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:52.495 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:52.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:52.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:52.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:52.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:52.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:52.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:52.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:53.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.048+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:53.051 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.048+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:53.082 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.080+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:53.083 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:53.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:53.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:53.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.092+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:53.944 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:19:53.945 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.944+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:53.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:53.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:53.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:54.008 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:54.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:54.008 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:54.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:54.008 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:54.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:54.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:54.512+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:19:54.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:54.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:54.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:54.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:54.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:54.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:54.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:54.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:54.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:54.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:54.615 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:19:55.530 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:19:55.531 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:19:55.595 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2660,"stamp":"2023-10-14T01:19:53.785079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28950752088,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2418573,"num_read_kb":9423515,"num_write":2377385,"num_write_kb":9509941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80536,"ondisk_log_size":80536,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23216608,"kb_used_data":19017352,"kb_used_omap":1502,"kb_used_meta":4197729,"kb_avail":259898912,"statfs":{"total":289910292480,"available":266136485888,"internally_reserved":0,"allocated":19473768448,"data_stored":23783764364,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":4298474961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,5,5,9,15,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22663168,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5531,"num_read_kb":22076,"num_write":5546,"num_write_kb":22184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002039"},"pg_stats":[{"pgid":"2.7","version":"150'319041","reported_seq":626381,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:52.645443+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:19:52.645443+0000","last_peered":"2023-10-14T01:19:52.645443+0000","last_clean":"2023-10-14T01:19:52.645443+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:19:52.645443+0000","last_undegraded":"2023-10-14T01:19:52.645443+0000","last_fullsized":"2023-10-14T01:19:52.645443+0000","mapping_epoch":13,"log_start":"146'309002","ondisk_log_start":"146'309002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":24,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3662352822,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":308346,"num_read_kb":1195843,"num_write":300440,"num_write_kb":1201518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"150'319340","reported_seq":627188,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:52.644585+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:19:52.644585+0000","last_peered":"2023-10-14T01:19:52.644585+0000","last_clean":"2023-10-14T01:19:52.644585+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:19:52.644585+0000","last_undegraded":"2023-10-14T01:19:52.644585+0000","last_fullsized":"2023-10-14T01:19:52.644585+0000","mapping_epoch":13,"log_start":"146'309313","ondisk_log_start":"146'309313","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1120,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 564s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3654099312,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":310239,"num_read_kb":1193445,"num_write":300189,"num_write_kb":1200383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"150'288494","reported_seq":574437,"reported_epoch":150,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:52.617836+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:19:52.617836+0000","last_peered":"2023-10-14T01:19:52.617836+0000","last_clean":"2023-10-14T01:19:52.617836+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:19:52.617836+0000","last_undegraded":"2023-10-14T01:19:52.617836+0000","last_fullsized":"2023-10-14T01:19:52.617836+0000","mapping_epoch":13,"log_start":"146'278405","ondisk_log_start":"146'278405","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":103,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 31s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3497136530,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":286341,"num_read_kb":1129416,"num_write":287818,"num_write_kb":1151242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"150'282136","reported_seq":564455,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:52.574476+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:19:52.574476+0000","last_peered":"2023-10-14T01:19:52.574476+0000","last_clean":"2023-10-14T01:19:52.574476+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:19:52.574476+0000","last_undegraded":"2023-10-14T01:19:52.574476+0000","last_fullsized":"2023-10-14T01:19:52.574476+0000","mapping_epoch":13,"log_start":"146'272107","ondisk_log_start":"146'272107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3421368451,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":282570,"num_read_kb":1112565,"num_write":280860,"num_write_kb":1123362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"150'318102","reported_seq":636976,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:53.645544+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:19:53.645544+0000","last_peered":"2023-10-14T01:19:53.645544+0000","last_clean":"2023-10-14T01:19:53.645544+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:19:53.645544+0000","last_undegraded":"2023-10-14T01:19:53.645544+0000","last_fullsized":"2023-10-14T01:19:53.645544+0000","mapping_epoch":13,"log_start":"146'308009","ondisk_log_start":"146'308009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3848511807,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":319240,"num_read_kb":1251070,"num_write":315639,"num_write_kb":1262409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"150'306526","reported_seq":608342,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:50.044338+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:19:50.044338+0000","last_peered":"2023-10-14T01:19:50.044338+0000","last_clean":"2023-10-14T01:19:50.044338+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:19:50.044338+0000","last_undegraded":"2023-10-14T01:19:50.044338+0000","last_fullsized":"2023-10-14T01:19:50.044338+0000","mapping_epoch":13,"log_start":"146'296509","ondisk_log_start":"146'296509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3610280205,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":302555,"num_read_kb":1173544,"num_write":297151,"num_write_kb":1188390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"150'311254","reported_seq":620364,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:50.064768+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:19:50.064768+0000","last_peered":"2023-10-14T01:19:50.064768+0000","last_clean":"2023-10-14T01:19:50.064768+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:19:50.064768+0000","last_undegraded":"2023-10-14T01:19:50.064768+0000","last_fullsized":"2023-10-14T01:19:50.064768+0000","mapping_epoch":13,"log_start":"146'301211","ondisk_log_start":"146'301211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3660603641,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":309952,"num_read_kb":1196920,"num_write":299906,"num_write_kb":1199350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"150'305581","reported_seq":604349,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:19:52.565323+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:19:52.565323+0000","last_peered":"2023-10-14T01:19:52.565323+0000","last_clean":"2023-10-14T01:19:52.565323+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:19:52.565323+0000","last_undegraded":"2023-10-14T01:19:52.565323+0000","last_fullsized":"2023-10-14T01:19:52.565323+0000","mapping_epoch":13,"log_start":"146'295509","ondisk_log_start":"146'295509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3595808952,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":299204,"num_read_kb":1170603,"num_write":295213,"num_write_kb":1180739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":361,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-10-14T01:17:59.733796+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:17:59.733796+0000","last_peered":"2023-10-14T01:17:59.733796+0000","last_clean":"2023-10-14T01:17:59.733796+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:17:59.733796+0000","last_undegraded":"2023-10-14T01:17:59.733796+0000","last_fullsized":"2023-10-14T01:17:59.733796+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28950161720,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2418447,"num_read_kb":9423406,"num_write":2377216,"num_write_kb":9507393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19472297984,"data_stored":23783776880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80409,"ondisk_log_size":80409,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4909884,"kb_used_data":3569204,"kb_used_omap":317,"kb_used_meta":1340354,"kb_avail":89461956,"statfs":{"total":96636764160,"available":91609042944,"internally_reserved":0,"allocated":3654864896,"data_stored":4463987634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1372523158},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.95599999999999996,"15min":0.82199999999999995},"min":{"1min":0.19700000000000001,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":6.2160000000000002,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.65800000000000003,"15min":0.78000000000000003},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19800000000000001},"max":{"1min":1.0429999999999999,"5min":7.6680000000000001,"15min":7.782},"last":0.33100000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.2370000000000001,"15min":1.3799999999999999},"min":{"1min":0.23200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.5489999999999999,"15min":1.4670000000000001},"min":{"1min":0.26500000000000001,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":7.431,"5min":8.8650000000000002,"15min":14.586},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739357,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10795512,"kb_used_data":9507952,"kb_used_omap":759,"kb_used_meta":1286792,"kb_avail":83576328,"statfs":{"total":96636764160,"available":85582159872,"internally_reserved":0,"allocated":9736142848,"data_stored":11891565644,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":1317675765},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,7,14,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.147,"15min":0.77400000000000002},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":4.0140000000000002,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.221},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.609,"15min":0.96399999999999997},"min":{"1min":0.217,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":3.98,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.254}]},{"osd":2,"last update":"Sat Oct 14 01:18:58 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3049999999999999,"15min":1.1970000000000001},"min":{"1min":0.26600000000000001,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":10.112,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.143,"15min":1.349},"min":{"1min":0.216,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.484,"5min":24.831,"15min":39.412999999999997},"last":0.35599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7511212,"kb_used_data":5940196,"kb_used_omap":425,"kb_used_meta":1570582,"kb_avail":86860628,"statfs":{"total":96636764160,"available":88945283072,"internally_reserved":0,"allocated":6082760704,"data_stored":7428211086,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1608276038},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.0920000000000001,"15min":0.91700000000000004},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":5.5220000000000002,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.54300000000000004,"15min":0.58499999999999996},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.17499999999999999},"max":{"1min":0.92700000000000005,"5min":4.069,"15min":9.2439999999999998},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:19:45 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.163,"15min":1.093},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.1970000000000001,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.60399999999999998,"15min":0.76400000000000001},"min":{"1min":0.246,"5min":0.223,"15min":0.223},"max":{"1min":1.5249999999999999,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6082379776,"data_stored":7428146810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9736364032,"data_stored":11892103480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3653554176,"data_stored":4463526590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:19:55.596 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:55.597 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:55.597 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:55.598 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:57.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.220+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:19:57.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:57.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:57.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:57.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:57.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:57.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:57.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.356+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:19:57.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:19:57.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:19:57.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:19:57.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:19:57.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:57.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:57.853 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:57.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:19:57.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:57.860+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:19:57.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:57.872+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:19:57.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:19:57.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:19:57.889 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:19:57.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:19:59.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:59.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:19:59.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:59.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:19:59.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:59.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:19:59.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:19:59.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:02.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:02.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:02.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:02.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:02.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:02.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:02.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:02.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:02.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:02.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:02.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:02.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:04.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:04.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:04.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:04.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:04.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:04.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:04.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:04.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:04.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:04.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:04.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:04.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:04.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:04.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:08.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:08.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.672+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:08.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:08.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:08.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:08.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.776+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:08.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:08.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:08.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:08.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:08.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:08.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:08.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:08.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:09.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:09.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:09.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:09.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:09.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:09.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:09.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:09.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:09.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:10.065 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:10.064+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:10.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:10.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:10.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:10.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:10.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:10.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:11.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:11.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:11.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:11.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:11.632 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:11.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:11.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:11.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:12.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:12.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:12.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:12.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:12.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:12.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:12.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:12.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:12.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:12.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:14.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.008+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:14.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:14.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:14.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:14.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.096+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:14.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:14.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:14.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:14.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:14.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:15.599 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:15.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:15.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:15.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:15.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:15.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:15.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:16.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:16.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:16.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:16.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:16.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:16.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:16.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:16.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:16.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:16.625 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:20:16.626 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:20:16.668 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2672,"stamp":"2023-10-14T01:20:15.788874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28985572184,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2426895,"num_read_kb":9456759,"num_write":2385911,"num_write_kb":9544045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80421,"ondisk_log_size":80421,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23413596,"kb_used_data":19085884,"kb_used_omap":1502,"kb_used_meta":4326177,"kb_avail":259701924,"statfs":{"total":289910292480,"available":265934770176,"internally_reserved":0,"allocated":19543945216,"data_stored":23853928919,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":4430005713},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,8,10,5,21,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16465920,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3851,"num_read_kb":15384,"num_write":4036,"num_write_kb":16144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001649"},"pg_stats":[{"pgid":"2.7","version":"151'320180","reported_seq":628564,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:13.212478+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:20:13.212478+0000","last_peered":"2023-10-14T01:20:13.212478+0000","last_clean":"2023-10-14T01:20:13.212478+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:20:13.212478+0000","last_undegraded":"2023-10-14T01:20:13.212478+0000","last_fullsized":"2023-10-14T01:20:13.212478+0000","mapping_epoch":13,"log_start":"147'310102","ondisk_log_start":"147'310102","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":60,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 38s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3666698678,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":309368,"num_read_kb":1199931,"num_write":301503,"num_write_kb":1205770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"151'320436","reported_seq":629272,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:13.176057+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:20:13.176057+0000","last_peered":"2023-10-14T01:20:13.176057+0000","last_clean":"2023-10-14T01:20:13.176057+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:20:13.176057+0000","last_undegraded":"2023-10-14T01:20:13.176057+0000","last_fullsized":"2023-10-14T01:20:13.176057+0000","mapping_epoch":13,"log_start":"147'310413","ondisk_log_start":"147'310413","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1154,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 585s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3658273136,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":311205,"num_read_kb":1197305,"num_write":301212,"num_write_kb":1204475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"151'289520","reported_seq":576500,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:13.106498+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:20:13.106498+0000","last_peered":"2023-10-14T01:20:13.106498+0000","last_clean":"2023-10-14T01:20:13.106498+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:20:13.106498+0000","last_undegraded":"2023-10-14T01:20:13.106498+0000","last_fullsized":"2023-10-14T01:20:13.106498+0000","mapping_epoch":13,"log_start":"146'279505","ondisk_log_start":"146'279505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":162,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 52s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3501171090,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":287366,"num_read_kb":1133516,"num_write":288810,"num_write_kb":1155210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"151'283087","reported_seq":566323,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:13.148491+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:20:13.148491+0000","last_peered":"2023-10-14T01:20:13.148491+0000","last_clean":"2023-10-14T01:20:13.148491+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:20:13.148491+0000","last_undegraded":"2023-10-14T01:20:13.148491+0000","last_fullsized":"2023-10-14T01:20:13.148491+0000","mapping_epoch":13,"log_start":"146'273007","ondisk_log_start":"146'273007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3425259651,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":283491,"num_read_kb":1116249,"num_write":281811,"num_write_kb":1127166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"151'319127","reported_seq":638977,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:13.930156+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:20:13.930156+0000","last_peered":"2023-10-14T01:20:13.930156+0000","last_clean":"2023-10-14T01:20:13.930156+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:20:13.930156+0000","last_undegraded":"2023-10-14T01:20:13.930156+0000","last_fullsized":"2023-10-14T01:20:13.930156+0000","mapping_epoch":13,"log_start":"147'309109","ondisk_log_start":"147'309109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3852673343,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":320204,"num_read_kb":1254914,"num_write":316662,"num_write_kb":1266501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"151'307721","reported_seq":610802,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:15.238684+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:20:15.238684+0000","last_peered":"2023-10-14T01:20:15.238684+0000","last_clean":"2023-10-14T01:20:15.238684+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:20:15.238684+0000","last_undegraded":"2023-10-14T01:20:15.238684+0000","last_fullsized":"2023-10-14T01:20:15.238684+0000","mapping_epoch":13,"log_start":"147'297709","ondisk_log_start":"147'297709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3615080717,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":303798,"num_read_kb":1178512,"num_write":298326,"num_write_kb":1193090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"151'312569","reported_seq":622919,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:15.245987+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:20:15.245987+0000","last_peered":"2023-10-14T01:20:15.245987+0000","last_clean":"2023-10-14T01:20:15.245987+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:20:15.245987+0000","last_undegraded":"2023-10-14T01:20:15.245987+0000","last_fullsized":"2023-10-14T01:20:15.245987+0000","mapping_epoch":13,"log_start":"147'302511","ondisk_log_start":"147'302511","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3665883385,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":311176,"num_read_kb":1201812,"num_write":301196,"num_write_kb":1204510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"151'306619","reported_seq":606352,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:13.114957+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:20:13.114957+0000","last_peered":"2023-10-14T01:20:13.114957+0000","last_clean":"2023-10-14T01:20:13.114957+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:20:13.114957+0000","last_undegraded":"2023-10-14T01:20:13.114957+0000","last_fullsized":"2023-10-14T01:20:13.114957+0000","mapping_epoch":13,"log_start":"147'296609","ondisk_log_start":"147'296609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3599941816,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":300161,"num_read_kb":1174411,"num_write":296222,"num_write_kb":1184775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":363,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-10-14T01:20:02.873977+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:20:02.873977+0000","last_peered":"2023-10-14T01:20:02.873977+0000","last_clean":"2023-10-14T01:20:02.873977+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:20:02.873977+0000","last_undegraded":"2023-10-14T01:20:02.873977+0000","last_fullsized":"2023-10-14T01:20:02.873977+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":28984981816,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2426769,"num_read_kb":9456650,"num_write":2385742,"num_write_kb":9541497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19540959232,"data_stored":23852438128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80294,"ondisk_log_size":80294,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739376,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4974612,"kb_used_data":3583948,"kb_used_omap":317,"kb_used_meta":1390338,"kb_avail":89397228,"statfs":{"total":96636764160,"available":91542761472,"internally_reserved":0,"allocated":3669962752,"data_stored":4479081419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1423706774},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,0,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.95599999999999996,"15min":0.82199999999999995},"min":{"1min":0.19700000000000001,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":6.2160000000000002,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":2.177},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.65800000000000003,"15min":0.78000000000000003},"min":{"1min":0.214,"5min":0.19900000000000001,"15min":0.19800000000000001},"max":{"1min":1.0429999999999999,"5min":7.6680000000000001,"15min":7.782},"last":0.28299999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.2370000000000001,"15min":1.3799999999999999},"min":{"1min":0.23200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":12.131,"5min":12.131,"15min":12.131},"last":14.005000000000001},{"interface":"front","average":{"1min":1.7050000000000001,"5min":1.5489999999999999,"15min":1.4670000000000001},"min":{"1min":0.26500000000000001,"5min":0.22800000000000001,"15min":0.20899999999999999},"max":{"1min":7.431,"5min":8.8650000000000002,"15min":14.586},"last":0.42299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739361,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10830580,"kb_used_data":9540904,"kb_used_omap":759,"kb_used_meta":1288904,"kb_avail":83541260,"statfs":{"total":96636764160,"available":85546250240,"internally_reserved":0,"allocated":9769885696,"data_stored":11925304421,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":1319838453},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,7,2,12,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.147,"15min":0.77400000000000002},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":4.0140000000000002,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.443},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.609,"15min":0.96399999999999997},"min":{"1min":0.217,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":3.98,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":1.284,"15min":1.1950000000000001},"min":{"1min":0.28100000000000003,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":7.5739999999999998,"5min":24.870000000000001,"15min":43.871000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.236,"15min":1.3700000000000001},"min":{"1min":0.215,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.6000000000000001,"5min":24.831,"15min":39.412999999999997},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7608404,"kb_used_data":5961032,"kb_used_omap":425,"kb_used_meta":1646934,"kb_avail":86763436,"statfs":{"total":96636764160,"available":88845758464,"internally_reserved":0,"allocated":6104096768,"data_stored":7449543079,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1686460486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.0920000000000001,"15min":0.91700000000000004},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":5.5220000000000002,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.54300000000000004,"15min":0.58499999999999996},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.17499999999999999},"max":{"1min":0.92700000000000005,"5min":4.069,"15min":9.2439999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:19:45 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.163,"15min":1.093},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.1970000000000001,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.377},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.60399999999999998,"15min":0.76400000000000001},"min":{"1min":0.246,"5min":0.223,"15min":0.223},"max":{"1min":1.5249999999999999,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6102761472,"data_stored":7448528506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9769484288,"data_stored":11925223736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3668713472,"data_stored":4478685886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:20:16.669 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:16.670 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:16.670 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:16.671 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:16.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:16.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:17.000 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:17.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:17.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:17.028+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:17.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:17.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:17.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:17.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:17.034 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:17.032+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:17.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:17.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:17.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:17.352+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:17.363 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:17.360+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:17.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:17.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:18.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:18.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:18.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:18.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:18.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:18.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.536+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:18.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:18.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:18.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:18.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:18.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:19.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:19.040+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:19.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:19.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:19.060 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:19.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:19.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:19.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:20.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:20.124+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:20.137 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:20.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:21.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:20:21.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:21.657 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:21.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:21.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:21.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:21.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:21.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.692+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:21.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:21.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:23.571 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:23.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:23.572 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:23.568+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:23.598 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:23.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:23.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:23.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:23.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:23.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:24.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:24.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:24.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:24.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:24.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:24.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.456+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:24.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:24.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:24.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:24.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:24.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:26.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:26.844+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:26.887 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:26.884+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:26.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:26.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:26.896 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:26.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:26.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:26.892+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:27.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:27.092+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:27.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:27.104+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:27.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:27.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:27.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:27.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:27.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:27.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:27.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:27.136+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:27.201 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:27.200+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:27.202 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:27.200+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:27.210 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:27.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:27.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:27.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:27.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:27.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:27.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:27.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:27.238 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:27.236+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:28.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:28.144+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:28.159 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:28.156+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:28.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:28.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:28.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:28.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:28.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:28.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:28.170 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:28.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:32.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:32.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:32.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:32.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:32.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:32.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:32.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:32.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:32.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:32.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:32.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:32.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:33.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:33.423+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:33.434 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:33.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:33.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:33.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:34.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:34.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:34.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:34.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:34.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:34.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:34.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:34.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:35.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:35.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:35.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:35.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:35.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:35.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:35.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:35.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:35.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:35.775+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:35.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:35.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:35.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:35.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:35.961 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:35.959+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:35.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:35.959+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:35.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:35.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:36.672 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:37.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:37.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:37.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:37.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:37.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:37.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:37.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:37.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:37.599 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:20:37.600 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:20:37.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:37.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:37.662 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2683,"stamp":"2023-10-14T01:20:35.803100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29018360664,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2434649,"num_read_kb":9487731,"num_write":2393946,"num_write_kb":9576185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80374,"ondisk_log_size":80374,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23774684,"kb_used_data":19150596,"kb_used_omap":1502,"kb_used_meta":4622561,"kb_avail":259340836,"statfs":{"total":289910292480,"available":265565016064,"internally_reserved":0,"allocated":19610210304,"data_stored":23920181665,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":4733502929},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,5,13,10,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16609280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3916,"num_read_kb":15644,"num_write":4071,"num_write_kb":16284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008741"},"pg_stats":[{"pgid":"2.7","version":"152'321314","reported_seq":630689,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:33.834782+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:20:33.834782+0000","last_peered":"2023-10-14T01:20:33.834782+0000","last_clean":"2023-10-14T01:20:33.834782+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:20:33.834782+0000","last_undegraded":"2023-10-14T01:20:33.834782+0000","last_fullsized":"2023-10-14T01:20:33.834782+0000","mapping_epoch":13,"log_start":"147'311302","ondisk_log_start":"147'311302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":114,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 59s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3670913462,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":310327,"num_read_kb":1203767,"num_write":302537,"num_write_kb":1209906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'321570","reported_seq":631459,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:33.857743+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:20:33.857743+0000","last_peered":"2023-10-14T01:20:33.857743+0000","last_clean":"2023-10-14T01:20:33.857743+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:20:33.857743+0000","last_undegraded":"2023-10-14T01:20:33.857743+0000","last_fullsized":"2023-10-14T01:20:33.857743+0000","mapping_epoch":13,"log_start":"147'311513","ondisk_log_start":"147'311513","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1174,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 605s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3662565744,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":312235,"num_read_kb":1201409,"num_write":302263,"num_write_kb":1208679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'290529","reported_seq":578482,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:33.811551+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:20:33.811551+0000","last_peered":"2023-10-14T01:20:33.811551+0000","last_clean":"2023-10-14T01:20:33.811551+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:20:33.811551+0000","last_undegraded":"2023-10-14T01:20:33.811551+0000","last_fullsized":"2023-10-14T01:20:33.811551+0000","mapping_epoch":13,"log_start":"147'280505","ondisk_log_start":"147'280505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":224,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 72s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3505152402,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":288331,"num_read_kb":1137368,"num_write":289785,"num_write_kb":1159110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'284038","reported_seq":568206,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:33.774684+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:20:33.774684+0000","last_peered":"2023-10-14T01:20:33.774684+0000","last_clean":"2023-10-14T01:20:33.774684+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:20:33.774684+0000","last_undegraded":"2023-10-14T01:20:33.774684+0000","last_fullsized":"2023-10-14T01:20:33.774684+0000","mapping_epoch":13,"log_start":"147'274007","ondisk_log_start":"147'274007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3429134467,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":284414,"num_read_kb":1119941,"num_write":282762,"num_write_kb":1130970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"152'320196","reported_seq":641063,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:34.130038+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:20:34.130038+0000","last_peered":"2023-10-14T01:20:34.130038+0000","last_clean":"2023-10-14T01:20:34.130038+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:20:34.130038+0000","last_undegraded":"2023-10-14T01:20:34.130038+0000","last_fullsized":"2023-10-14T01:20:34.130038+0000","mapping_epoch":13,"log_start":"147'310109","ondisk_log_start":"147'310109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3857031487,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":321218,"num_read_kb":1258962,"num_write":317729,"num_write_kb":1270769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"152'308713","reported_seq":612751,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:35.291137+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:20:35.291137+0000","last_peered":"2023-10-14T01:20:35.291137+0000","last_clean":"2023-10-14T01:20:35.291137+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:20:35.291137+0000","last_undegraded":"2023-10-14T01:20:35.291137+0000","last_fullsized":"2023-10-14T01:20:35.291137+0000","mapping_epoch":13,"log_start":"147'298709","ondisk_log_start":"147'298709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3619012877,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":304737,"num_read_kb":1182264,"num_write":299290,"num_write_kb":1196946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"152'313616","reported_seq":624920,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:35.279643+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:20:35.279643+0000","last_peered":"2023-10-14T01:20:35.279643+0000","last_clean":"2023-10-14T01:20:35.279643+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:20:35.279643+0000","last_undegraded":"2023-10-14T01:20:35.279643+0000","last_fullsized":"2023-10-14T01:20:35.279643+0000","mapping_epoch":13,"log_start":"147'303611","ondisk_log_start":"147'303611","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3670016249,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":312104,"num_read_kb":1205520,"num_write":302208,"num_write_kb":1208558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"152'307636","reported_seq":608375,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:33.795882+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:20:33.795882+0000","last_peered":"2023-10-14T01:20:33.795882+0000","last_clean":"2023-10-14T01:20:33.795882+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:20:33.795882+0000","last_undegraded":"2023-10-14T01:20:33.795882+0000","last_fullsized":"2023-10-14T01:20:33.795882+0000","mapping_epoch":13,"log_start":"147'297609","ondisk_log_start":"147'297609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3603943608,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":301157,"num_read_kb":1178391,"num_write":297203,"num_write_kb":1188699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":364,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:09.830788+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:20:09.830788+0000","last_peered":"2023-10-14T01:20:09.830788+0000","last_clean":"2023-10-14T01:20:09.830788+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:20:09.830788+0000","last_undegraded":"2023-10-14T01:20:09.830788+0000","last_fullsized":"2023-10-14T01:20:09.830788+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29017770296,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2434523,"num_read_kb":9487622,"num_write":2393777,"num_write_kb":9573637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19606663168,"data_stored":23918142064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80247,"ondisk_log_size":80247,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739380,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5036352,"kb_used_data":3595704,"kb_used_omap":317,"kb_used_meta":1440322,"kb_avail":89335488,"statfs":{"total":96636764160,"available":91479539712,"internally_reserved":0,"allocated":3682000896,"data_stored":4491115449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1474890390},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.98999999999999999,"15min":0.83399999999999996},"min":{"1min":0.218,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":7.4669999999999996,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.73499999999999999,"15min":0.78600000000000003},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":7.3339999999999996,"5min":7.6680000000000001,"15min":7.782},"last":0.29199999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:20:31 2023","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.4630000000000001,"15min":1.3380000000000001},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":14.005000000000001,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.161,"5min":1.6100000000000001,"15min":1.454},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":7.5529999999999999,"5min":8.8650000000000002,"15min":14.586},"last":0.252}]}]},{"osd":1,"up_from":8,"seq":34359739365,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11046096,"kb_used_data":9574280,"kb_used_omap":759,"kb_used_meta":1471048,"kb_avail":83325744,"statfs":{"total":96636764160,"available":85325561856,"internally_reserved":0,"allocated":9804062720,"data_stored":11959477331,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":1506353909},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,6,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":1.147,"15min":0.77400000000000002},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":4.0140000000000002,"5min":23.434000000000001,"15min":23.434000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":1.609,"15min":0.96399999999999997},"min":{"1min":0.217,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":3.98,"5min":24.908000000000001,"15min":24.908000000000001},"last":0.36099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":1.284,"15min":1.1950000000000001},"min":{"1min":0.28100000000000003,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":7.5739999999999998,"5min":24.870000000000001,"15min":43.871000000000002},"last":5.1589999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.236,"15min":1.3700000000000001},"min":{"1min":0.215,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.6000000000000001,"5min":24.831,"15min":39.412999999999997},"last":5.2949999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739378,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7692236,"kb_used_data":5980612,"kb_used_omap":425,"kb_used_meta":1711190,"kb_avail":86679604,"statfs":{"total":96636764160,"available":88759914496,"internally_reserved":0,"allocated":6124146688,"data_stored":7469588885,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1752258630},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.0920000000000001,"15min":0.91700000000000004},"min":{"1min":0.20799999999999999,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":5.5220000000000002,"5min":11.734999999999999,"15min":11.734999999999999},"last":3.9009999999999998},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.54300000000000004,"15min":0.58499999999999996},"min":{"1min":0.218,"5min":0.20000000000000001,"15min":0.17499999999999999},"max":{"1min":0.92700000000000005,"5min":4.069,"15min":9.2439999999999998},"last":0.46700000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:19:45 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":1.163,"15min":1.093},"min":{"1min":0.23200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.1970000000000001,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.60399999999999998,"15min":0.76400000000000001},"min":{"1min":0.246,"5min":0.223,"15min":0.223},"max":{"1min":1.5249999999999999,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6123163648,"data_stored":7468930682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9802752000,"data_stored":11958491448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3680747520,"data_stored":4490719934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:20:37.664 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:37.664 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:37.665 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:37.665 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:39.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:39.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:40.677 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:40.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:41.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:41.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:41.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:41.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:41.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:41.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:43.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:43.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:43.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:43.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:43.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:43.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:43.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:43.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:43.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:43.839+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:44.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:44.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:44.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:44.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:44.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:20:44.433 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:20:44.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:20:44.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:44.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.491+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:44.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:44.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.499+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:44.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:44.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:44.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:44.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:44.929 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:44.927+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:44.965 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:44.963+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:44.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:44.991+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:45.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:44.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:45.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:44.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:45.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:44.999+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:45.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.231+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:20:45.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:45.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:45.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:45.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:45.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:45.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:45.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:45.247+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:47.677 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:47.675+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:47.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:47.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:47.693 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:47.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:47.694 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:47.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:48.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:48.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:48.650 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:48.647+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:48.681 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:48.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:48.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:48.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:48.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:48.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:48.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:48.691+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:48.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:48.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:48.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:48.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:48.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:48.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:48.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:48.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:48.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:48.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:48.888 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:48.887+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:49.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:49.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:49.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:49.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:49.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:49.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:49.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:49.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:54.928 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:54.927+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:54.976 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:54.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:55.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:54.995+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:55.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:55.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:55.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:55.011+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:55.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:55.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:55.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:55.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:55.388 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:55.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:55.411 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.407+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:55.424 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.423+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:55.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:55.450 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:55.451 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:55.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:56.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:56.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:56.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:56.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:20:56.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:56.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:56.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:56.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:56.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:56.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:56.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:56.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:56.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:56.879+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:57.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:57.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.051+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:57.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:57.099 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.095+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:57.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:57.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:20:57.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.111+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:20:57.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:57.530 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.523+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:57.531 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:57.527+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:20:57.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:57.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:57.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:57.651+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:57.667 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:58.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:58.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:58.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:58.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:58.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:58.370 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:58.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:58.371 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:58.367+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:20:58.385 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:58.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:20:58.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:58.383+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:20:58.523 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:20:58.523 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:20:58.574 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2694,"stamp":"2023-10-14T01:20:57.809781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29051501400,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2442507,"num_read_kb":9519103,"num_write":2402079,"num_write_kb":9608717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80395,"ondisk_log_size":80395,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24039840,"kb_used_data":19214024,"kb_used_omap":1502,"kb_used_meta":4824289,"kb_avail":259075680,"statfs":{"total":289910292480,"available":265293496320,"internally_reserved":0,"allocated":19675160576,"data_stored":23985131937,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":4940072401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,10,9,6,6,18,23,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16613376,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3887,"num_read_kb":15520,"num_write":4075,"num_write_kb":16300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002155"},"pg_stats":[{"pgid":"2.7","version":"152'322417","reported_seq":632864,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:54.816622+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:20:54.816622+0000","last_peered":"2023-10-14T01:20:54.816622+0000","last_clean":"2023-10-14T01:20:54.816622+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:20:54.816622+0000","last_undegraded":"2023-10-14T01:20:54.816622+0000","last_fullsized":"2023-10-14T01:20:54.816622+0000","mapping_epoch":13,"log_start":"148'312402","ondisk_log_start":"148'312402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":161,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 80s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3675111862,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":311375,"num_read_kb":1207955,"num_write":303566,"num_write_kb":1214022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'322651","reported_seq":633526,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:54.815751+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:20:54.815751+0000","last_peered":"2023-10-14T01:20:54.815751+0000","last_clean":"2023-10-14T01:20:54.815751+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:20:54.815751+0000","last_undegraded":"2023-10-14T01:20:54.815751+0000","last_fullsized":"2023-10-14T01:20:54.815751+0000","mapping_epoch":13,"log_start":"148'312613","ondisk_log_start":"148'312613","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1216,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 626s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3666669936,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":313193,"num_read_kb":1205229,"num_write":303273,"num_write_kb":1212719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'291582","reported_seq":580490,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:54.765282+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:20:54.765282+0000","last_peered":"2023-10-14T01:20:54.765282+0000","last_clean":"2023-10-14T01:20:54.765282+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:20:54.765282+0000","last_undegraded":"2023-10-14T01:20:54.765282+0000","last_fullsized":"2023-10-14T01:20:54.765282+0000","mapping_epoch":13,"log_start":"147'281505","ondisk_log_start":"147'281505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":272,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 93s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3509338514,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":289279,"num_read_kb":1141148,"num_write":290810,"num_write_kb":1163210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'285029","reported_seq":570153,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:54.754513+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:20:54.754513+0000","last_peered":"2023-10-14T01:20:54.754513+0000","last_clean":"2023-10-14T01:20:54.754513+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:20:54.754513+0000","last_undegraded":"2023-10-14T01:20:54.754513+0000","last_fullsized":"2023-10-14T01:20:54.754513+0000","mapping_epoch":13,"log_start":"147'275007","ondisk_log_start":"147'275007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3433160835,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":285370,"num_read_kb":1123757,"num_write":283753,"num_write_kb":1134934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"152'321249","reported_seq":643187,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:54.343947+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:20:54.343947+0000","last_peered":"2023-10-14T01:20:54.343947+0000","last_clean":"2023-10-14T01:20:54.343947+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:20:54.343947+0000","last_undegraded":"2023-10-14T01:20:54.343947+0000","last_fullsized":"2023-10-14T01:20:54.343947+0000","mapping_epoch":13,"log_start":"147'311209","ondisk_log_start":"147'311209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3861315903,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":322287,"num_read_kb":1263234,"num_write":318781,"num_write_kb":1274977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"152'309743","reported_seq":614721,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:55.494227+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:20:55.494227+0000","last_peered":"2023-10-14T01:20:55.494227+0000","last_clean":"2023-10-14T01:20:55.494227+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:20:55.494227+0000","last_undegraded":"2023-10-14T01:20:55.494227+0000","last_fullsized":"2023-10-14T01:20:55.494227+0000","mapping_epoch":13,"log_start":"147'299709","ondisk_log_start":"147'299709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3623043341,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":305663,"num_read_kb":1185960,"num_write":300280,"num_write_kb":1200906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"152'314730","reported_seq":627016,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:55.413969+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:20:55.413969+0000","last_peered":"2023-10-14T01:20:55.413969+0000","last_clean":"2023-10-14T01:20:55.413969+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:20:55.413969+0000","last_undegraded":"2023-10-14T01:20:55.413969+0000","last_fullsized":"2023-10-14T01:20:55.413969+0000","mapping_epoch":13,"log_start":"147'304711","ondisk_log_start":"147'304711","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3674366201,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":313077,"num_read_kb":1209404,"num_write":303272,"num_write_kb":1212814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"152'308632","reported_seq":610359,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:54.715886+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:20:54.715886+0000","last_peered":"2023-10-14T01:20:54.715886+0000","last_clean":"2023-10-14T01:20:54.715886+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:20:54.715886+0000","last_undegraded":"2023-10-14T01:20:54.715886+0000","last_fullsized":"2023-10-14T01:20:54.715886+0000","mapping_epoch":13,"log_start":"147'298609","ondisk_log_start":"147'298609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3607904440,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":302137,"num_read_kb":1182307,"num_write":298175,"num_write_kb":1192587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":364,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:09.830788+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:20:09.830788+0000","last_peered":"2023-10-14T01:20:09.830788+0000","last_clean":"2023-10-14T01:20:09.830788+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:20:09.830788+0000","last_undegraded":"2023-10-14T01:20:09.830788+0000","last_fullsized":"2023-10-14T01:20:09.830788+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29050911032,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2442381,"num_read_kb":9518994,"num_write":2401910,"num_write_kb":9606169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19673042944,"data_stored":23984521840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80268,"ondisk_log_size":80268,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739384,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5080160,"kb_used_data":3607576,"kb_used_omap":317,"kb_used_meta":1472258,"kb_avail":89291680,"statfs":{"total":96636764160,"available":91434680320,"internally_reserved":0,"allocated":3694157824,"data_stored":4503272377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1507592854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.98999999999999999,"15min":0.83399999999999996},"min":{"1min":0.218,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":7.4669999999999996,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.254},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.73499999999999999,"15min":0.78600000000000003},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":7.3339999999999996,"5min":7.6680000000000001,"15min":7.782},"last":0.38100000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:20:31 2023","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.4630000000000001,"15min":1.3380000000000001},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":14.005000000000001,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.161,"5min":1.6100000000000001,"15min":1.454},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":7.5529999999999999,"5min":8.8650000000000002,"15min":14.586},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739369,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11189632,"kb_used_data":9606264,"kb_used_omap":759,"kb_used_meta":1582600,"kb_avail":83182208,"statfs":{"total":96636764160,"available":85178580992,"internally_reserved":0,"allocated":9836814336,"data_stored":11992228947,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":1620583157},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,9,3,2,3,9,13,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.79200000000000004,"15min":0.76600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":0.44600000000000001,"5min":10.02,"15min":23.434000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.92200000000000004,"15min":0.94999999999999996},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.46999999999999997,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.313}]},{"osd":2,"last update":"Sat Oct 14 01:19:59 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":1.284,"15min":1.1950000000000001},"min":{"1min":0.28100000000000003,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":7.5739999999999998,"5min":24.870000000000001,"15min":43.871000000000002},"last":2.0179999999999998},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.236,"15min":1.3700000000000001},"min":{"1min":0.215,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.6000000000000001,"5min":24.831,"15min":39.412999999999997},"last":0.22800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7770048,"kb_used_data":6000184,"kb_used_omap":425,"kb_used_meta":1769430,"kb_avail":86601792,"statfs":{"total":96636764160,"available":88680235008,"internally_reserved":0,"allocated":6144188416,"data_stored":7489630613,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1811896390},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,1,2,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.1319999999999999,"15min":0.89700000000000002},"min":{"1min":0.223,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":3.9009999999999998,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.505,"15min":0.56699999999999995},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":0.72999999999999998,"5min":3.2530000000000001,"15min":9.2439999999999998},"last":0.247}]},{"osd":2,"last update":"Sat Oct 14 01:20:45 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":1.016,"15min":1.0129999999999999},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.0670000000000002,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.56599999999999995,"15min":0.74199999999999999},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.79000000000000004,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.51000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6143455232,"data_stored":7489222266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9836572672,"data_stored":11992312120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3693015040,"data_stored":4502987454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:20:58.576 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:58.576 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:58.577 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:58.577 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:58.855 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.851+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:58.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:58.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:58.902 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:58.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:58.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:58.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:58.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:59.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:20:59.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:20:59.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:59.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:59.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:59.287 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.283+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:20:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:20:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:20:59.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:20:59.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:20:59.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:59.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:20:59.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:20:59.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:00.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:00.015+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:01.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.247+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:01.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:01.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:01.269 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.263+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:01.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:01.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:01.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:01.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:01.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:01.739 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:01.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:02.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:02.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:02.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:02.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:02.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:02.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:02.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:02.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:03.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:03.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:03.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:03.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:03.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:03.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:03.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:03.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:03.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:03.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:03.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:03.907+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:05.576 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:05.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:05.577 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:05.575+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:05.588 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:05.587+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:05.614 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:05.611+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:05.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:05.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:05.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:05.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:05.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:05.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:05.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:05.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:05.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:05.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:05.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:05.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:05.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:05.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:05.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:05.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:07.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:07.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:07.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:07.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:07.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:07.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:09.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:09.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:09.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:09.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:09.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:09.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:09.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:09.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:09.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:09.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:10.443 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:10.439+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:10.457 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:10.455+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:10.489 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:10.487+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:10.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:10.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:10.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:10.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:10.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:10.503+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:11.012 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.011+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:11.029 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:11.045 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.043+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:11.064 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.059+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:11.064 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.059+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:11.065 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.059+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:11.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.351+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:11.365 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.363+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:11.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.387+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:11.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:11.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:11.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:11.391+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:11.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:11.684 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:11.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:11.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:11.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.783+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:11.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:11.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:11.836 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:11.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:11.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:12.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:12.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:12.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:12.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:12.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:12.755+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:12.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:12.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:12.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:12.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:14.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:14.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.299+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:14.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:14.324 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:14.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:14.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:14.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:14.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:14.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:14.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:14.962 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:14.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:14.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:14.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:15.024 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.019+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:21:15.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:15.037 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.035+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:15.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.043+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:15.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:15.074 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:15.074 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:15.074 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:15.071+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:15.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:15.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:21:15.417 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:15.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:15.418 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:15.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:15.421 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:15.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:15.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:15.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:15.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:15.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:16.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:16.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:16.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:16.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:16.596 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:16.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:16.872 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:16.871+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:16.888 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:16.887+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:16.892 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:16.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:16.907 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:16.903+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:16.908 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:16.903+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:16.908 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:16.903+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:17.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:17.759+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:17.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:17.763+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:17.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:17.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:17.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:17.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:17.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:17.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:17.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:17.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:17.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:17.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:18.578 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:19.298 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:21:19.298 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:21:19.342 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2704,"stamp":"2023-10-14T01:21:17.813120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29083614040,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2450408,"num_read_kb":9550643,"num_write":2409950,"num_write_kb":9640201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80576,"ondisk_log_size":80576,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24392364,"kb_used_data":19278548,"kb_used_omap":1502,"kb_used_meta":5112289,"kb_avail":258723156,"statfs":{"total":289910292480,"available":264932511744,"internally_reserved":0,"allocated":19741233152,"data_stored":24051204513,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":5234984401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,12,18,21,24,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15917056,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3967,"num_read_kb":15820,"num_write":3902,"num_write_kb":15608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002042"},"pg_stats":[{"pgid":"2.7","version":"152'323545","reported_seq":635018,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:15.469763+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:21:15.469763+0000","last_peered":"2023-10-14T01:21:15.469763+0000","last_clean":"2023-10-14T01:21:15.469763+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:21:15.469763+0000","last_undegraded":"2023-10-14T01:21:15.469763+0000","last_fullsized":"2023-10-14T01:21:15.469763+0000","mapping_epoch":13,"log_start":"149'313502","ondisk_log_start":"149'313502","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":201,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 101s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3679261110,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":312382,"num_read_kb":1211979,"num_write":304586,"num_write_kb":1218102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"152'323781","reported_seq":635656,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:15.464240+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:21:15.464240+0000","last_peered":"2023-10-14T01:21:15.464240+0000","last_clean":"2023-10-14T01:21:15.464240+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:21:15.464240+0000","last_undegraded":"2023-10-14T01:21:15.464240+0000","last_fullsized":"2023-10-14T01:21:15.464240+0000","mapping_epoch":13,"log_start":"149'313713","ondisk_log_start":"149'313713","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1240,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 647s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3670892912,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":314178,"num_read_kb":1209165,"num_write":304304,"num_write_kb":1216843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"152'292585","reported_seq":582472,"reported_epoch":152,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:15.435755+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:21:15.435755+0000","last_peered":"2023-10-14T01:21:15.435755+0000","last_clean":"2023-10-14T01:21:15.435755+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:21:15.435755+0000","last_undegraded":"2023-10-14T01:21:15.435755+0000","last_fullsized":"2023-10-14T01:21:15.435755+0000","mapping_epoch":13,"log_start":"148'282505","ondisk_log_start":"148'282505","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":341,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 114s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3513250194,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":290243,"num_read_kb":1144984,"num_write":291770,"num_write_kb":1167050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"152'285964","reported_seq":572013,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:21:15.447195+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:21:15.447195+0000","last_peered":"2023-10-14T01:21:15.447195+0000","last_clean":"2023-10-14T01:21:15.447195+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:21:15.447195+0000","last_undegraded":"2023-10-14T01:21:15.447195+0000","last_fullsized":"2023-10-14T01:21:15.447195+0000","mapping_epoch":13,"log_start":"147'275907","ondisk_log_start":"147'275907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3436957827,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":286295,"num_read_kb":1127445,"num_write":284688,"num_write_kb":1138674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"152'322338","reported_seq":645333,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:21:14.727823+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:21:14.727823+0000","last_peered":"2023-10-14T01:21:14.727823+0000","last_clean":"2023-10-14T01:21:14.727823+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:21:14.727823+0000","last_undegraded":"2023-10-14T01:21:14.727823+0000","last_fullsized":"2023-10-14T01:21:14.727823+0000","mapping_epoch":13,"log_start":"149'312309","ondisk_log_start":"149'312309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3865751871,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":323337,"num_read_kb":1267426,"num_write":319865,"num_write_kb":1279313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"152'310704","reported_seq":616666,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:21:15.599953+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:21:15.599953+0000","last_peered":"2023-10-14T01:21:15.599953+0000","last_clean":"2023-10-14T01:21:15.599953+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:21:15.599953+0000","last_undegraded":"2023-10-14T01:21:15.599953+0000","last_fullsized":"2023-10-14T01:21:15.599953+0000","mapping_epoch":13,"log_start":"148'300609","ondisk_log_start":"148'300609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3626795277,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":306633,"num_read_kb":1189840,"num_write":301193,"num_write_kb":1204558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"152'315755","reported_seq":629082,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:21:15.634834+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:21:15.634834+0000","last_peered":"2023-10-14T01:21:15.634834+0000","last_clean":"2023-10-14T01:21:15.634834+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:21:15.634834+0000","last_undegraded":"2023-10-14T01:21:15.634834+0000","last_fullsized":"2023-10-14T01:21:15.634834+0000","mapping_epoch":13,"log_start":"149'305711","ondisk_log_start":"149'305711","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3678298361,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":314092,"num_read_kb":1213452,"num_write":304237,"num_write_kb":1216674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"152'309642","reported_seq":612358,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:21:15.454227+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:21:15.454227+0000","last_peered":"2023-10-14T01:21:15.454227+0000","last_clean":"2023-10-14T01:21:15.454227+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:21:15.454227+0000","last_undegraded":"2023-10-14T01:21:15.454227+0000","last_fullsized":"2023-10-14T01:21:15.454227+0000","mapping_epoch":13,"log_start":"148'299609","ondisk_log_start":"148'299609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3611816120,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":303122,"num_read_kb":1186243,"num_write":299138,"num_write_kb":1196439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":364,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-14T01:20:09.830788+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:20:09.830788+0000","last_peered":"2023-10-14T01:20:09.830788+0000","last_clean":"2023-10-14T01:20:09.830788+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:20:09.830788+0000","last_undegraded":"2023-10-14T01:20:09.830788+0000","last_fullsized":"2023-10-14T01:20:09.830788+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29083023672,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2450282,"num_read_kb":9550534,"num_write":2409781,"num_write_kb":9637653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19737378816,"data_stored":24048857712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80449,"ondisk_log_size":80449,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739388,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5127852,"kb_used_data":3618724,"kb_used_omap":317,"kb_used_meta":1508802,"kb_avail":89243988,"statfs":{"total":96636764160,"available":91385843712,"internally_reserved":0,"allocated":3705573376,"data_stored":4514687929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1545013910},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,7,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":0.98999999999999999,"15min":0.83399999999999996},"min":{"1min":0.218,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":7.4669999999999996,"5min":7.6440000000000001,"15min":7.6440000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.73499999999999999,"15min":0.78600000000000003},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":7.3339999999999996,"5min":7.6680000000000001,"15min":7.782},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:20:31 2023","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.4630000000000001,"15min":1.3380000000000001},"min":{"1min":0.23000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":14.005000000000001,"5min":14.005000000000001,"15min":14.005000000000001},"last":6.4050000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.6100000000000001,"15min":1.454},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":7.5529999999999999,"5min":8.8650000000000002,"15min":14.586},"last":0.58899999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11393012,"kb_used_data":9639020,"kb_used_omap":759,"kb_used_meta":1753224,"kb_avail":82978828,"statfs":{"total":96636764160,"available":84970319872,"internally_reserved":0,"allocated":9870356480,"data_stored":12025771091,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":1795302133},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,12,6,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.79200000000000004,"15min":0.76600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":0.44600000000000001,"5min":10.02,"15min":23.434000000000001},"last":0.311},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.92200000000000004,"15min":0.94999999999999996},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.46999999999999997,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.70399999999999996}]},{"osd":2,"last update":"Sat Oct 14 01:21:00 2023","interfaces":[{"interface":"back","average":{"1min":2.9889999999999999,"5min":1.405,"15min":1.3540000000000001},"min":{"1min":0.28699999999999998,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":27.837,"5min":27.837,"15min":43.871000000000002},"last":2.5550000000000002},{"interface":"front","average":{"1min":2.7149999999999999,"5min":1.421,"15min":1.482},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.16800000000000001},"max":{"1min":25.710000000000001,"5min":25.710000000000001,"15min":39.412999999999997},"last":0.60499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739386,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7871500,"kb_used_data":6020804,"kb_used_omap":425,"kb_used_meta":1850262,"kb_avail":86500340,"statfs":{"total":96636764160,"available":88576348160,"internally_reserved":0,"allocated":6165303296,"data_stored":7510745493,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1894668358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.1319999999999999,"15min":0.89700000000000002},"min":{"1min":0.223,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":3.9009999999999998,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.505,"15min":0.56699999999999995},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":0.72999999999999998,"5min":3.2530000000000001,"15min":9.2439999999999998},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:20:45 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":1.016,"15min":1.0129999999999999},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.0670000000000002,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.56599999999999995,"15min":0.74199999999999999},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.79000000000000004,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.29499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6163828736,"data_stored":7509595770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9869033472,"data_stored":12024772920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3704516608,"data_stored":4514489022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:21:19.343 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:19.344 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:19.344 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:19.344 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:21:22.092 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:22.091+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:22.110 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:22.107+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:22.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:22.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:22.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:22.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:22.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:22.131+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:23.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:23.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:23.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:23.331+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:23.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:23.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:24.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:24.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:24.567 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:24.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:24.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:24.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:24.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:24.603+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:24.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:24.623+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:26.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:26.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:26.260 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:26.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:26.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:26.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:26.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:26.259+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:26.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:26.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:26.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:26.635+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:26.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:26.643+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:26.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:26.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:26.680 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:26.679+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:27.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:27.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:27.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:27.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:27.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:27.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:27.163 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:27.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:27.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:27.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:27.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:27.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:27.181 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:27.178+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:28.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:28.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:28.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:28.502+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:28.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:28.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:29.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:29.610+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:29.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:29.618+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:29.630 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:29.626+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:29.638 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:29.634+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:29.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:29.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:29.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:29.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:29.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:29.642+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:30.735 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:30.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:30.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:30.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:30.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:30.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:30.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:30.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:30.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:30.754+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:30.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:30.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:31.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:31.298+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:31.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:31.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:31.336 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:31.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:31.337 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:31.334+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:32.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:32.050+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:32.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:32.054+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:33.906 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:21:33.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:21:33.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:33.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:33.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:33.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:33.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:33.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:33.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:33.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:33.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:34.135 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:34.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.146+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:34.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:34.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:34.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:34.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:34.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:34.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:34.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:36.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:36.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:36.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:36.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:36.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:36.166+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:36.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:36.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:36.694 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:36.690+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:36.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:36.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:36.715 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:36.710+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:36.725 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:36.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:36.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:36.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:36.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:36.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:36.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:36.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:37.273 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:37.270+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:37.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:37.270+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:37.282 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:37.278+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:37.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:37.290+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:37.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:37.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:37.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:37.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:37.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:37.302+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:38.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:38.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:38.215 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:38.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:38.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:38.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:38.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:38.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:38.918 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:38.906+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:38.937 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:38.934+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:38.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:38.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:38.943 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:38.938+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:39.119 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:39.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:39.164 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:39.162+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:39.187 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:39.182+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:39.346 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:40.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:40.714+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:40.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:40.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:40.738 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:40.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:40.750 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:40.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:40.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:40.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:40.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:40.746+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:40.831 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:21:40.832 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:21:40.893 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2716,"stamp":"2023-10-14T01:21:39.820131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29116517208,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2458361,"num_read_kb":9582403,"num_write":2418039,"num_write_kb":9672557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80496,"ondisk_log_size":80496,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24689376,"kb_used_data":19342088,"kb_used_omap":1502,"kb_used_meta":5345761,"kb_avail":258426144,"statfs":{"total":289910292480,"available":264628371456,"internally_reserved":0,"allocated":19806298112,"data_stored":24116257002,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":5474059729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,16,11,16,19,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16359424,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3969,"num_read_kb":15848,"num_write":4015,"num_write_kb":16060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.020284"},"pg_stats":[{"pgid":"2.7","version":"153'324644","reported_seq":637153,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:36.051690+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:21:36.051690+0000","last_peered":"2023-10-14T01:21:36.051690+0000","last_clean":"2023-10-14T01:21:36.051690+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:21:36.051690+0000","last_undegraded":"2023-10-14T01:21:36.051690+0000","last_fullsized":"2023-10-14T01:21:36.051690+0000","mapping_epoch":13,"log_start":"149'314602","ondisk_log_start":"149'314602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":229,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 121s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3683492278,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":313393,"num_read_kb":1216003,"num_write":305625,"num_write_kb":1222258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'324859","reported_seq":637755,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:36.048931+0000","last_change":"2023-10-14T01:10:28.110621+0000","last_active":"2023-10-14T01:21:36.048931+0000","last_peered":"2023-10-14T01:21:36.048931+0000","last_clean":"2023-10-14T01:21:36.048931+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:21:36.048931+0000","last_undegraded":"2023-10-14T01:21:36.048931+0000","last_fullsized":"2023-10-14T01:21:36.048931+0000","mapping_epoch":13,"log_start":"149'314813","ondisk_log_start":"149'314813","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1279,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 667s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3675038064,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":315166,"num_read_kb":1213113,"num_write":305317,"num_write_kb":1220895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'293637","reported_seq":584506,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:36.047741+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:21:36.047741+0000","last_peered":"2023-10-14T01:21:36.047741+0000","last_clean":"2023-10-14T01:21:36.047741+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:21:36.047741+0000","last_undegraded":"2023-10-14T01:21:36.047741+0000","last_fullsized":"2023-10-14T01:21:36.047741+0000","mapping_epoch":13,"log_start":"149'283606","ondisk_log_start":"149'283606","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":379,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 134s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3517440402,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":291217,"num_read_kb":1148876,"num_write":292802,"num_write_kb":1171178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'286874","reported_seq":573878,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:21:36.052463+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:21:36.052463+0000","last_peered":"2023-10-14T01:21:36.052463+0000","last_clean":"2023-10-14T01:21:36.052463+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:21:36.052463+0000","last_undegraded":"2023-10-14T01:21:36.052463+0000","last_fullsized":"2023-10-14T01:21:36.052463+0000","mapping_epoch":13,"log_start":"149'276807","ondisk_log_start":"149'276807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3440648323,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":287250,"num_read_kb":1131253,"num_write":285598,"num_write_kb":1142314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"153'323439","reported_seq":647536,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:21:34.962050+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:21:34.962050+0000","last_peered":"2023-10-14T01:21:34.962050+0000","last_clean":"2023-10-14T01:21:34.962050+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:21:34.962050+0000","last_undegraded":"2023-10-14T01:21:34.962050+0000","last_fullsized":"2023-10-14T01:21:34.962050+0000","mapping_epoch":13,"log_start":"149'313409","ondisk_log_start":"149'313409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3870245183,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":324433,"num_read_kb":1271810,"num_write":320964,"num_write_kb":1283709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"153'311747","reported_seq":618671,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:21:35.665207+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:21:35.665207+0000","last_peered":"2023-10-14T01:21:35.665207+0000","last_clean":"2023-10-14T01:21:35.665207+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:21:35.665207+0000","last_undegraded":"2023-10-14T01:21:35.665207+0000","last_fullsized":"2023-10-14T01:21:35.665207+0000","mapping_epoch":13,"log_start":"149'301709","ondisk_log_start":"149'301709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3630911757,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":307580,"num_read_kb":1193628,"num_write":302208,"num_write_kb":1208618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"153'316810","reported_seq":631171,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:21:35.692075+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:21:35.692075+0000","last_peered":"2023-10-14T01:21:35.692075+0000","last_clean":"2023-10-14T01:21:35.692075+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:21:35.692075+0000","last_undegraded":"2023-10-14T01:21:35.692075+0000","last_fullsized":"2023-10-14T01:21:35.692075+0000","mapping_epoch":13,"log_start":"149'306711","ondisk_log_start":"149'306711","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3682431225,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":315104,"num_read_kb":1217492,"num_write":305257,"num_write_kb":1220754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"153'310625","reported_seq":614322,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:21:36.054860+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:21:36.054860+0000","last_peered":"2023-10-14T01:21:36.054860+0000","last_clean":"2023-10-14T01:21:36.054860+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:21:36.054860+0000","last_undegraded":"2023-10-14T01:21:36.054860+0000","last_fullsized":"2023-10-14T01:21:36.054860+0000","mapping_epoch":13,"log_start":"149'300609","ondisk_log_start":"149'300609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3615719608,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":304092,"num_read_kb":1190119,"num_write":300099,"num_write_kb":1200283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":365,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:29.815656+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:20:29.815656+0000","last_peered":"2023-10-14T01:20:29.815656+0000","last_clean":"2023-10-14T01:20:29.815656+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:20:29.815656+0000","last_undegraded":"2023-10-14T01:20:29.815656+0000","last_fullsized":"2023-10-14T01:20:29.815656+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29115926840,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2458235,"num_read_kb":9582294,"num_write":2417870,"num_write_kb":9670009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":7986,"num_omap_keys":180,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19803590656,"data_stored":24115069552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80369,"ondisk_log_size":80369,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739392,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5171580,"kb_used_data":3630516,"kb_used_omap":317,"kb_used_meta":1540738,"kb_avail":89200260,"statfs":{"total":96636764160,"available":91341066240,"internally_reserved":0,"allocated":3717648384,"data_stored":4526758780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1577716374},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75900000000000001,"15min":0.84699999999999998},"min":{"1min":0.19400000000000001,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":6.4589999999999996,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.53900000000000003,"15min":0.753},"min":{"1min":0.247,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.47799999999999998,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.42099999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:21:32 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.3109999999999999,"15min":1.3120000000000001},"min":{"1min":0.23599999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":6.4050000000000002,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.25},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.4890000000000001,"15min":1.4259999999999999},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":9.9329999999999998,"5min":9.9329999999999998,"15min":14.586},"last":1.9470000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739377,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11532088,"kb_used_data":9670960,"kb_used_omap":759,"kb_used_meta":1860360,"kb_avail":82839752,"statfs":{"total":96636764160,"available":84827906048,"internally_reserved":0,"allocated":9903063040,"data_stored":12058473494,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":1905009397},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,11,9,8,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:39 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.79200000000000004,"15min":0.76600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":0.44600000000000001,"5min":10.02,"15min":23.434000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.92200000000000004,"15min":0.94999999999999996},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":0.46999999999999997,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:21:00 2023","interfaces":[{"interface":"back","average":{"1min":2.9889999999999999,"5min":1.405,"15min":1.3540000000000001},"min":{"1min":0.28699999999999998,"5min":0.23599999999999999,"15min":0.222},"max":{"1min":27.837,"5min":27.837,"15min":43.871000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":2.7149999999999999,"5min":1.421,"15min":1.482},"min":{"1min":0.22800000000000001,"5min":0.20599999999999999,"15min":0.16800000000000001},"max":{"1min":25.710000000000001,"5min":25.710000000000001,"15min":39.412999999999997},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739390,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7985708,"kb_used_data":6040612,"kb_used_omap":425,"kb_used_meta":1944662,"kb_avail":86386132,"statfs":{"total":96636764160,"available":88459399168,"internally_reserved":0,"allocated":6185586688,"data_stored":7531024728,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":1991333958},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,7,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.1319999999999999,"15min":0.89700000000000002},"min":{"1min":0.223,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":3.9009999999999998,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.505,"15min":0.56699999999999995},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":0.72999999999999998,"5min":3.2530000000000001,"15min":9.2439999999999998},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:20:45 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":1.016,"15min":1.0129999999999999},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.0670000000000002,"5min":9.8059999999999992,"15min":10.539999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.56599999999999995,"15min":0.74199999999999999},"min":{"1min":0.254,"5min":0.223,"15min":0.223},"max":{"1min":0.79000000000000004,"5min":4.4260000000000002,"15min":8.9710000000000001},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6184468480,"data_stored":7530235514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9902444544,"data_stored":12058183992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3716677632,"data_stored":4526650046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:21:40.894 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:40.895 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:40.895 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:40.895 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:21:40.933 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:40.930+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:40.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:40.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:40.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:40.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:40.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:40.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:40.968 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:40.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:40.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:40.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:40.981 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:40.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:42.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:42.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:42.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:42.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:42.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:42.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:42.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:42.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:43.013 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:43.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:43.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:43.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:43.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:43.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:43.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:43.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:43.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:43.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:43.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:43.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:43.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:43.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:44.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:44.638+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:44.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:44.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:44.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:44.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:44.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:44.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:44.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:44.646+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:45.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:45.418+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:46.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:46.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:46.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:46.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:46.829 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:46.826+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:47.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:47.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:47.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:47.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:47.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:47.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:47.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:47.402+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:49.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:49.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:49.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:49.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:49.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:49.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:49.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:49.466+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:49.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:49.478+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:49.485 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:49.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:49.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:49.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:49.486 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:49.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:49.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:49.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:49.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:49.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:49.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:49.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:49.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:49.634+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:50.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:50.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.242+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:50.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.250+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:50.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:50.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:50.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:50.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.526+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:50.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:50.535 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:50.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:50.559 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:50.571 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:50.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:52.972 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:52.966+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:52.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:52.966+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:52.993 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:52.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:52.993 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:52.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:52.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:52.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:52.994 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:52.990+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:53.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:21:53.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:53.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:21:53.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:53.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:53.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.794+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:53.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:53.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:53.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:54.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:54.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:21:54.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:54.558+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:21:54.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:54.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:21:54.696 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:54.694+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:54.709 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:54.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:54.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:54.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:54.710 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:54.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:55.806 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:55.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:21:55.806 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:55.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:21:55.812 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:55.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:21:55.813 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:55.810+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:21:55.821 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:55.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:21:55.838 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:55.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:21:55.868 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:21:55.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:21:59.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:59.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:21:59.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:59.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:21:59.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:59.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:21:59.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:21:59.118+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:00.897 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:01.865 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:22:01.866 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:22:01.956 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2727,"stamp":"2023-10-14T01:22:01.829404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29157387096,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2468194,"num_read_kb":9621619,"num_write":2428063,"num_write_kb":9712653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80651,"ondisk_log_size":80651,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25132976,"kb_used_data":19422936,"kb_used_omap":1502,"kb_used_meta":5708513,"kb_avail":257982544,"statfs":{"total":289910292480,"available":264174125056,"internally_reserved":0,"allocated":19889086464,"data_stored":24199045354,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1538607,"internal_metadata":5845517777},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,8,17,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24932352,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5945,"num_read_kb":23704,"num_write":6113,"num_write_kb":24452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004396"},"pg_stats":[{"pgid":"2.7","version":"153'325992","reported_seq":639814,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:01.808099+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:22:01.808099+0000","last_peered":"2023-10-14T01:22:01.808099+0000","last_clean":"2023-10-14T01:22:01.808099+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:22:01.808099+0000","last_undegraded":"2023-10-14T01:22:01.808099+0000","last_fullsized":"2023-10-14T01:22:01.808099+0000","mapping_epoch":13,"log_start":"149'315902","ondisk_log_start":"149'315902","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":283,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 147s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3688526262,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":314670,"num_read_kb":1221095,"num_write":306861,"num_write_kb":1227202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'326256","reported_seq":640458,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:01.797547+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:22:01.797547+0000","last_peered":"2023-10-14T01:22:01.797547+0000","last_clean":"2023-10-14T01:22:01.797547+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:22:01.797547+0000","last_undegraded":"2023-10-14T01:22:01.797547+0000","last_fullsized":"2023-10-14T01:22:01.797547+0000","mapping_epoch":13,"log_start":"149'316213","ondisk_log_start":"149'316213","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3680334192,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":316447,"num_read_kb":1218221,"num_write":306616,"num_write_kb":1226091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'294885","reported_seq":586951,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:01.760177+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:22:01.760177+0000","last_peered":"2023-10-14T01:22:01.760177+0000","last_clean":"2023-10-14T01:22:01.760177+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:22:01.760177+0000","last_undegraded":"2023-10-14T01:22:01.760177+0000","last_fullsized":"2023-10-14T01:22:01.760177+0000","mapping_epoch":13,"log_start":"149'284806","ondisk_log_start":"149'284806","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":470,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 160s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3522318738,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":292401,"num_read_kb":1153592,"num_write":294000,"num_write_kb":1175970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'288098","reported_seq":576277,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:01.746168+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:22:01.746168+0000","last_peered":"2023-10-14T01:22:01.746168+0000","last_clean":"2023-10-14T01:22:01.746168+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:22:01.746168+0000","last_undegraded":"2023-10-14T01:22:01.746168+0000","last_fullsized":"2023-10-14T01:22:01.746168+0000","mapping_epoch":13,"log_start":"149'278007","ondisk_log_start":"149'278007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3445657731,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":288423,"num_read_kb":1135941,"num_write":286822,"num_write_kb":1147210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"153'324783","reported_seq":650139,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:00.089254+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:22:00.089254+0000","last_peered":"2023-10-14T01:22:00.089254+0000","last_clean":"2023-10-14T01:22:00.089254+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:22:00.089254+0000","last_undegraded":"2023-10-14T01:22:00.089254+0000","last_fullsized":"2023-10-14T01:22:00.089254+0000","mapping_epoch":13,"log_start":"149'314709","ondisk_log_start":"149'314709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3875721535,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":325692,"num_read_kb":1276842,"num_write":322305,"num_write_kb":1289073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"153'313006","reported_seq":621103,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:00.901757+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:22:00.901757+0000","last_peered":"2023-10-14T01:22:00.901757+0000","last_clean":"2023-10-14T01:22:00.901757+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:22:00.901757+0000","last_undegraded":"2023-10-14T01:22:00.901757+0000","last_fullsized":"2023-10-14T01:22:00.901757+0000","mapping_epoch":13,"log_start":"149'302909","ondisk_log_start":"149'302909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3635802381,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":308740,"num_read_kb":1198248,"num_write":303411,"num_write_kb":1213430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"153'318125","reported_seq":633779,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:00.905274+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:22:00.905274+0000","last_peered":"2023-10-14T01:22:00.905274+0000","last_clean":"2023-10-14T01:22:00.905274+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:22:00.905274+0000","last_undegraded":"2023-10-14T01:22:00.905274+0000","last_fullsized":"2023-10-14T01:22:00.905274+0000","mapping_epoch":13,"log_start":"149'308111","ondisk_log_start":"149'308111","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3687497977,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":316373,"num_read_kb":1222544,"num_write":306502,"num_write_kb":1225734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"153'311945","reported_seq":616882,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:01.721707+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:22:01.721707+0000","last_peered":"2023-10-14T01:22:01.721707+0000","last_clean":"2023-10-14T01:22:01.721707+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:22:01.721707+0000","last_undegraded":"2023-10-14T01:22:01.721707+0000","last_fullsized":"2023-10-14T01:22:01.721707+0000","mapping_epoch":13,"log_start":"149'301909","ondisk_log_start":"149'301909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3620937912,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":305322,"num_read_kb":1195027,"num_write":301377,"num_write_kb":1205395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":365,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-10-14T01:20:29.815656+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:20:29.815656+0000","last_peered":"2023-10-14T01:20:29.815656+0000","last_clean":"2023-10-14T01:20:29.815656+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:20:29.815656+0000","last_undegraded":"2023-10-14T01:20:29.815656+0000","last_fullsized":"2023-10-14T01:20:29.815656+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29156796728,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2468068,"num_read_kb":9621510,"num_write":2427894,"num_write_kb":9710105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19885473792,"data_stored":24196952688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33904,"internal_metadata":0},"log_size":80524,"ondisk_log_size":80524,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739397,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5264856,"kb_used_data":3645776,"kb_used_omap":317,"kb_used_meta":1618754,"kb_avail":89106984,"statfs":{"total":96636764160,"available":91245551616,"internally_reserved":0,"allocated":3733274624,"data_stored":4542385020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1657604758},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75900000000000001,"15min":0.84699999999999998},"min":{"1min":0.19400000000000001,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":6.4589999999999996,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.53900000000000003,"15min":0.753},"min":{"1min":0.247,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.47799999999999998,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.22700000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:21:32 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.3109999999999999,"15min":1.3120000000000001},"min":{"1min":0.23599999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":6.4050000000000002,"5min":14.005000000000001,"15min":14.005000000000001},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.4890000000000001,"15min":1.4259999999999999},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":9.9329999999999998,"5min":9.9329999999999998,"15min":14.586},"last":0.90100000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739382,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11781604,"kb_used_data":9712092,"kb_used_omap":759,"kb_used_meta":2068744,"kb_avail":82590236,"statfs":{"total":96636764160,"available":84572401664,"internally_reserved":0,"allocated":9945182208,"data_stored":12100592662,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":777483,"internal_metadata":2118394613},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,7,14,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72799999999999998,"15min":0.79000000000000004},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":3.6619999999999999,"5min":10.02,"15min":23.434000000000001},"last":0.249},{"interface":"front","average":{"1min":1.016,"5min":0.91600000000000004,"15min":0.96799999999999997},"min":{"1min":0.221,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":5.649,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:22:00 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.423,"15min":1.254},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23499999999999999},"max":{"1min":3.484,"5min":27.837,"15min":27.837},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.2809999999999999,"15min":1.3480000000000001},"min":{"1min":0.22700000000000001,"5min":0.20599999999999999,"15min":0.16800000000000001},"max":{"1min":3.234,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.40600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739395,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8086516,"kb_used_data":6065068,"kb_used_omap":425,"kb_used_meta":2021014,"kb_avail":86285324,"statfs":{"total":96636764160,"available":88356171776,"internally_reserved":0,"allocated":6210629632,"data_stored":7556067672,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":436154,"internal_metadata":2069518406},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0640000000000001,"15min":0.91000000000000003},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":5.3360000000000003,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.52800000000000002,"15min":0.58199999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.1819999999999999,"5min":3.2530000000000001,"15min":9.2439999999999998},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.82199999999999995,"15min":0.97199999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.7349999999999999,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.71899999999999997},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":3.6789999999999998,"5min":3.7770000000000001,"15min":8.9710000000000001},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6209826816,"data_stored":7555593850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10538,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9943822336,"data_stored":12099561784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3731824640,"data_stored":4541797054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:22:01.958 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:01.959 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:01.959 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:02.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.234+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:02.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.266+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:02.380 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:02.370+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:02.386 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:02.378+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:02.402 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:02.398+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:02.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:02.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:02.407 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:02.402+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:02.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:02.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:02.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:02.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:02.602 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:02.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:02.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:02.598+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:03.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:03.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:03.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:03.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:03.534 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:03.530+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:04.741 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:04.738+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:04.753 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:04.750+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:04.761 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:04.758+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:04.768 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:04.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:04.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:04.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:04.769 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:04.766+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:05.766 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:22:05.762+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:22:06.762 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:06.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:07.112 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:07.110+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:07.124 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:07.118+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:07.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:07.122+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:07.135 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:07.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:07.136 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:07.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:07.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:07.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:07.536 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:07.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:07.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:07.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:08.448 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:08.438+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:10.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:22:10.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.226+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:22:10.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:10.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:10.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:10.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:10.258 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.254+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:10.267 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:10.277 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:10.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:10.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:12.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:12.434+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:12.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:12.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:12.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:12.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:12.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:12.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:12.494 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:12.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:12.695 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:12.694+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:12.696 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:12.694+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:12.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:12.694+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:12.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:12.698+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:12.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:12.714+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:12.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:12.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:12.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:12.722+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:13.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:13.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:13.061 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:13.058+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:13.134 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:13.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:13.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:13.174+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:13.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:13.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:13.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:13.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:13.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:13.202+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:15.298 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:15.294+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:15.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:15.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:15.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:15.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:15.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:15.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:15.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:15.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:15.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:15.354+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:18.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:18.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:18.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:18.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:18.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:18.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:18.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:18.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:18.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:18.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:18.145 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:18.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:18.146 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:18.142+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:18.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:18.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.138+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:18.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.174+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:18.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:18.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:18.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:18.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:18.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:18.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:18.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:19.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:19.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:19.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:19.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:19.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:19.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:19.808 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:19.802+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:19.823 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:19.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:19.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:19.854+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:19.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:19.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:19.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:19.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:19.879 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:19.874+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:20.350 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:20.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:20.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:20.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:20.368 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:20.362+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:20.389 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:20.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:20.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:20.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:20.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:20.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:20.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:20.386+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:21.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:21.593+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:21.618 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:21.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:21.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:21.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:21.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:21.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:21.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:21.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:21.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:21.637+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:21.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:21.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:21.961 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:22.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:22.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:22.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:22.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:22.088 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:22.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:22.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:22.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:22.392 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:22.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.397+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:22.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:22.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:22.427 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:22.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:22.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:22.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:22.961 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:22:22.962 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:22:23.012 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2737,"stamp":"2023-10-14T01:22:21.836370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29185448792,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2475048,"num_read_kb":9648947,"num_write":2434925,"num_write_kb":9740101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80449,"ondisk_log_size":80449,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23064840,"kb_used_data":19479656,"kb_used_omap":1635,"kb_used_meta":3583516,"kb_avail":260050680,"statfs":{"total":289910292480,"available":266291896320,"internally_reserved":0,"allocated":19947167744,"data_stored":24257126634,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":3669520449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,12,11,9,8,18,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19562496,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4807,"num_read_kb":19172,"num_write":4783,"num_write_kb":19132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"2.7","version":"153'326826","reported_seq":641444,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:17.216227+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:22:17.216227+0000","last_peered":"2023-10-14T01:22:17.216227+0000","last_clean":"2023-10-14T01:22:17.216227+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:22:17.216227+0000","last_undegraded":"2023-10-14T01:22:17.216227+0000","last_fullsized":"2023-10-14T01:22:17.216227+0000","mapping_epoch":13,"log_start":"150'316802","ondisk_log_start":"150'316802","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":297,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 162s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3691774390,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":315451,"num_read_kb":1224211,"num_write":307652,"num_write_kb":1230366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"153'327058","reported_seq":642042,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:17.207600+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:22:17.207600+0000","last_peered":"2023-10-14T01:22:17.207600+0000","last_clean":"2023-10-14T01:22:17.207600+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:22:17.207600+0000","last_undegraded":"2023-10-14T01:22:17.207600+0000","last_fullsized":"2023-10-14T01:22:17.207600+0000","mapping_epoch":13,"log_start":"150'317013","ondisk_log_start":"150'317013","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3683475824,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":317214,"num_read_kb":1221277,"num_write":307382,"num_write_kb":1229155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"153'295663","reported_seq":588473,"reported_epoch":153,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:17.205340+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:22:17.205340+0000","last_peered":"2023-10-14T01:22:17.205340+0000","last_clean":"2023-10-14T01:22:17.205340+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:22:17.205340+0000","last_undegraded":"2023-10-14T01:22:17.205340+0000","last_fullsized":"2023-10-14T01:22:17.205340+0000","mapping_epoch":13,"log_start":"149'285606","ondisk_log_start":"149'285606","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":510,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 176s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3525407122,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":293135,"num_read_kb":1156516,"num_write":294756,"num_write_kb":1178994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"153'288827","reported_seq":577739,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:17.205847+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:22:17.205847+0000","last_peered":"2023-10-14T01:22:17.205847+0000","last_clean":"2023-10-14T01:22:17.205847+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:22:17.205847+0000","last_undegraded":"2023-10-14T01:22:17.205847+0000","last_fullsized":"2023-10-14T01:22:17.205847+0000","mapping_epoch":13,"log_start":"149'278807","ondisk_log_start":"149'278807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3448639619,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":289157,"num_read_kb":1138865,"num_write":287551,"num_write_kb":1150126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"153'325879","reported_seq":652321,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:20.492120+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:22:20.492120+0000","last_peered":"2023-10-14T01:22:20.492120+0000","last_clean":"2023-10-14T01:22:20.492120+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:22:20.492120+0000","last_undegraded":"2023-10-14T01:22:20.492120+0000","last_fullsized":"2023-10-14T01:22:20.492120+0000","mapping_epoch":13,"log_start":"150'315809","ondisk_log_start":"150'315809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3880165695,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":326769,"num_read_kb":1281142,"num_write":323392,"num_write_kb":1293421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"153'314043","reported_seq":623171,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:21.102289+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:22:21.102289+0000","last_peered":"2023-10-14T01:22:21.102289+0000","last_clean":"2023-10-14T01:22:21.102289+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:22:21.102289+0000","last_undegraded":"2023-10-14T01:22:21.102289+0000","last_fullsized":"2023-10-14T01:22:21.102289+0000","mapping_epoch":13,"log_start":"149'304009","ondisk_log_start":"149'304009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3639836941,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":309750,"num_read_kb":1202280,"num_write":304400,"num_write_kb":1217386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"153'319157","reported_seq":635833,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:21.058910+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:22:21.058910+0000","last_peered":"2023-10-14T01:22:21.058910+0000","last_clean":"2023-10-14T01:22:21.058910+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:22:21.058910+0000","last_undegraded":"2023-10-14T01:22:21.058910+0000","last_fullsized":"2023-10-14T01:22:21.058910+0000","mapping_epoch":13,"log_start":"150'309111","ondisk_log_start":"150'309111","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3691487481,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":317374,"num_read_kb":1226540,"num_write":307474,"num_write_kb":1229622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"153'312735","reported_seq":618430,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:17.193701+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:22:17.193701+0000","last_peered":"2023-10-14T01:22:17.193701+0000","last_clean":"2023-10-14T01:22:17.193701+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:22:17.193701+0000","last_undegraded":"2023-10-14T01:22:17.193701+0000","last_fullsized":"2023-10-14T01:22:17.193701+0000","mapping_epoch":13,"log_start":"149'302709","ondisk_log_start":"149'302709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3624071352,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":306072,"num_read_kb":1198007,"num_write":302149,"num_write_kb":1208483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":366,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:21:29.814136+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:21:29.814136+0000","last_peered":"2023-10-14T01:21:29.814136+0000","last_clean":"2023-10-14T01:21:29.814136+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:21:29.814136+0000","last_undegraded":"2023-10-14T01:21:29.814136+0000","last_fullsized":"2023-10-14T01:21:29.814136+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29184858424,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2474922,"num_read_kb":9648838,"num_write":2434756,"num_write_kb":9737553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19943501824,"data_stored":24254980720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80322,"ondisk_log_size":80322,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739401,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5290104,"kb_used_data":3657136,"kb_used_omap":317,"kb_used_meta":1632642,"kb_avail":89081736,"statfs":{"total":96636764160,"available":91219697664,"internally_reserved":0,"allocated":3744907264,"data_stored":4554017660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1671826070},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,2,4,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.75900000000000001,"15min":0.84699999999999998},"min":{"1min":0.19400000000000001,"5min":0.19,"15min":0.16300000000000001},"max":{"1min":6.4589999999999996,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.53900000000000003,"15min":0.753},"min":{"1min":0.247,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":0.47799999999999998,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.246}]},{"osd":1,"last update":"Sat Oct 14 01:21:32 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.3109999999999999,"15min":1.3120000000000001},"min":{"1min":0.23599999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":6.4050000000000002,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.4890000000000001,"15min":1.4259999999999999},"min":{"1min":0.252,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":9.9329999999999998,"5min":9.9329999999999998,"15min":14.586},"last":0.56200000000000006}]}]},{"osd":1,"up_from":8,"seq":34359739385,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10935544,"kb_used_data":9736616,"kb_used_omap":822,"kb_used_meta":1198089,"kb_avail":83436296,"statfs":{"total":96636764160,"available":85438767104,"internally_reserved":0,"allocated":9970294784,"data_stored":12125705238,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":1226843394},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,4,5,1,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.72799999999999998,"15min":0.79000000000000004},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":3.6619999999999999,"5min":10.02,"15min":23.434000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.016,"5min":0.91600000000000004,"15min":0.96799999999999997},"min":{"1min":0.221,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":5.649,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:22:00 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.423,"15min":1.254},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23499999999999999},"max":{"1min":3.484,"5min":27.837,"15min":27.837},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.2809999999999999,"15min":1.3480000000000001},"min":{"1min":0.22700000000000001,"5min":0.20599999999999999,"15min":0.16800000000000001},"max":{"1min":3.234,"5min":25.710000000000001,"15min":25.710000000000001},"last":4.9340000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739399,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6839192,"kb_used_data":6085904,"kb_used_omap":495,"kb_used_meta":752784,"kb_avail":87532648,"statfs":{"total":96636764160,"available":89633431552,"internally_reserved":0,"allocated":6231965696,"data_stored":7577403736,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":770850985},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0640000000000001,"15min":0.91000000000000003},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":5.3360000000000003,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.52800000000000002,"15min":0.58199999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.1819999999999999,"5min":3.2530000000000001,"15min":9.2439999999999998},"last":0.27600000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.82199999999999995,"15min":0.97199999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.7349999999999999,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.71899999999999997},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":3.6789999999999998,"5min":3.7770000000000001,"15min":8.9710000000000001},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6230560768,"data_stored":7576327802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9968984064,"data_stored":12124723512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3743956992,"data_stored":4553929406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:22:23.014 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:23.014 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:23.015 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:23.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:23.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:23.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:23.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:23.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:23.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:23.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:23.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:24.173 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:24.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:24.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:24.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:24.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:24.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:24.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:24.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:24.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:24.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:24.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:24.717+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:25.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:25.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:25.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:25.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:25.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:25.958 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:25.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:25.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:25.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:25.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:26.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:26.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:26.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:26.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:26.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:26.796 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:26.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:26.801 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:26.797+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:26.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.809+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:26.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:26.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:26.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:26.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:26.813+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:26.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:26.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:26.826 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:26.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:26.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:26.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:26.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.821+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:26.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:26.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:26.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:27.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:27.333+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:27.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:27.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:27.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:27.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:27.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:27.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:27.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:27.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:28.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.053+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:28.621 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:28.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:28.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:28.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:28.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:28.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:28.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:28.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:28.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:28.884 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:28.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:28.885 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:28.881+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:28.925 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:28.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:28.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:28.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:28.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:28.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:28.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:28.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:28.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:28.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:29.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.261+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:22:29.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:29.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.297+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:29.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:29.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:29.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.313+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:29.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:29.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:29.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:29.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:30.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:30.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:30.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:30.210 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:30.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.209+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:30.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.221+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:30.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:30.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:30.285+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:34.610 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:34.601+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:34.622 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:34.617+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:34.654 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:34.649+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:34.655 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:34.653+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:34.664 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:34.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:34.664 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:34.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:35.161 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:35.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:35.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:35.240 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:35.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:35.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.253+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:35.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:35.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:35.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:35.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:35.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:35.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:35.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:35.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:35.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:37.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.349+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:37.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.365+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:37.384 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:37.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:37.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:22:37.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:37.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:37.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:37.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:37.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:37.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:37.883 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:22:37.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:22:37.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:37.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:37.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:37.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:37.927 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:37.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.925+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:37.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:37.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:38.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:38.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:38.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:38.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:38.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:38.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:38.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:38.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:38.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:38.919 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:38.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:38.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:38.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:38.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:38.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:38.920 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:38.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:38.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:38.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:38.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:38.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:38.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:38.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:39.921 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.917+0000 7feb91e47640 -1 reset not still connected to 0x5632e174fc70 2023-10-14T01:22:39.931 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.925+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:22:39.939 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:22:39.939 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:39.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:39.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:39.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:39.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:39.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:39.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:40.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:40.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:40.872 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:40.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:40.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:40.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.889+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:40.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:40.950 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:40.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:41.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:41.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:41.298 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:41.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:41.299 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:41.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:41.299 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:41.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:42.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:42.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:42.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:42.537+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:42.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:42.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:42.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:42.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:42.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:42.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:42.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:42.561+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:42.572 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:42.569+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:42.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:42.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:42.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:42.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:42.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:42.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:42.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:42.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:42.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:42.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:42.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:42.577+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:43.016 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:43.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.245+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:43.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.257+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:43.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:43.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:43.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:43.576 INFO:teuthology.orchestra.run.smithi138.stderr:2023-10-14T01:22:43.573+0000 7f2d577fe640 0 --2- 172.21.15.138:0/3577662499 >> [v2:172.21.15.138:3300/0,v1:172.21.15.138:6789/0] conn(0x7f2d40003a40 0x7f2d5811c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-14T01:22:43.812 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:43.809+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:43.825 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:43.821+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:43.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:43.833+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:43.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:43.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:43.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:43.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:43.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:43.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:43.876 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:43.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.881+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:43.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:43.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:43.897+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:44.123 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:22:44.124 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:22:44.180 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2749,"stamp":"2023-10-14T01:22:43.844138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29224368984,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2484864,"num_read_kb":9688079,"num_write":2444450,"num_write_kb":9778201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80475,"ondisk_log_size":80475,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23438540,"kb_used_data":19552612,"kb_used_omap":1635,"kb_used_meta":3884252,"kb_avail":259676980,"statfs":{"total":289910292480,"available":265909227520,"internally_reserved":0,"allocated":20021874688,"data_stored":24331820978,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":3977474113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,4,6,6,13,16,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17252352,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4452,"num_read_kb":17764,"num_write":4228,"num_write_kb":16912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001819"},"pg_stats":[{"pgid":"2.7","version":"154'328326","reported_seq":644317,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:42.965877+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:22:42.965877+0000","last_peered":"2023-10-14T01:22:42.965877+0000","last_clean":"2023-10-14T01:22:42.965877+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:22:42.965877+0000","last_undegraded":"2023-10-14T01:22:42.965877+0000","last_fullsized":"2023-10-14T01:22:42.965877+0000","mapping_epoch":13,"log_start":"150'318302","ondisk_log_start":"150'318302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":334,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 188s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3697336758,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":316792,"num_read_kb":1229555,"num_write":309013,"num_write_kb":1235810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'328451","reported_seq":644807,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:22:42.991016+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:22:42.991016+0000","last_peered":"2023-10-14T01:22:42.991016+0000","last_clean":"2023-10-14T01:22:42.991016+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:22:42.991016+0000","last_undegraded":"2023-10-14T01:22:42.991016+0000","last_fullsized":"2023-10-14T01:22:42.991016+0000","mapping_epoch":13,"log_start":"150'318413","ondisk_log_start":"150'318413","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3688661360,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":318565,"num_read_kb":1226677,"num_write":308656,"num_write_kb":1234251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'296998","reported_seq":591130,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:42.989403+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:22:42.989403+0000","last_peered":"2023-10-14T01:22:42.989403+0000","last_clean":"2023-10-14T01:22:42.989403+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:22:42.989403+0000","last_undegraded":"2023-10-14T01:22:42.989403+0000","last_fullsized":"2023-10-14T01:22:42.989403+0000","mapping_epoch":13,"log_start":"150'286906","ondisk_log_start":"150'286906","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":566,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 201s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3530723730,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":294446,"num_read_kb":1161732,"num_write":296058,"num_write_kb":1184202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'290045","reported_seq":580240,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:22:42.979608+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:22:42.979608+0000","last_peered":"2023-10-14T01:22:42.979608+0000","last_clean":"2023-10-14T01:22:42.979608+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:22:42.979608+0000","last_undegraded":"2023-10-14T01:22:42.979608+0000","last_fullsized":"2023-10-14T01:22:42.979608+0000","mapping_epoch":13,"log_start":"150'280007","ondisk_log_start":"150'280007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3453599875,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":290443,"num_read_kb":1143989,"num_write":288769,"num_write_kb":1154998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"154'326928","reported_seq":654542,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:22:40.684196+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:22:40.684196+0000","last_peered":"2023-10-14T01:22:40.684196+0000","last_clean":"2023-10-14T01:22:40.684196+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:22:40.684196+0000","last_undegraded":"2023-10-14T01:22:40.684196+0000","last_fullsized":"2023-10-14T01:22:40.684196+0000","mapping_epoch":13,"log_start":"150'316909","ondisk_log_start":"150'316909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3884425535,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":327937,"num_read_kb":1285802,"num_write":324436,"num_write_kb":1297597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"154'315108","reported_seq":625278,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:22:41.299296+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:22:41.299296+0000","last_peered":"2023-10-14T01:22:41.299296+0000","last_clean":"2023-10-14T01:22:41.299296+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:22:41.299296+0000","last_undegraded":"2023-10-14T01:22:41.299296+0000","last_fullsized":"2023-10-14T01:22:41.299296+0000","mapping_epoch":13,"log_start":"150'305009","ondisk_log_start":"150'305009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3644113165,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":310778,"num_read_kb":1206364,"num_write":305445,"num_write_kb":1221566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"154'320241","reported_seq":637958,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:22:41.221141+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:22:41.221141+0000","last_peered":"2023-10-14T01:22:41.221141+0000","last_clean":"2023-10-14T01:22:41.221141+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:22:41.221141+0000","last_undegraded":"2023-10-14T01:22:41.221141+0000","last_fullsized":"2023-10-14T01:22:41.221141+0000","mapping_epoch":13,"log_start":"150'310211","ondisk_log_start":"150'310211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3695825145,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":318395,"num_read_kb":1230616,"num_write":308531,"num_write_kb":1233850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"154'314017","reported_seq":621031,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:22:42.967889+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:22:42.967889+0000","last_peered":"2023-10-14T01:22:42.967889+0000","last_clean":"2023-10-14T01:22:42.967889+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:22:42.967889+0000","last_undegraded":"2023-10-14T01:22:42.967889+0000","last_fullsized":"2023-10-14T01:22:42.967889+0000","mapping_epoch":13,"log_start":"150'304009","ondisk_log_start":"150'304009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3629093048,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":307382,"num_read_kb":1203235,"num_write":303373,"num_write_kb":1213379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":367,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:06.897931+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:22:06.897931+0000","last_peered":"2023-10-14T01:22:06.897931+0000","last_clean":"2023-10-14T01:22:06.897931+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:22:06.897931+0000","last_undegraded":"2023-10-14T01:22:06.897931+0000","last_fullsized":"2023-10-14T01:22:06.897931+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29223778616,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2484738,"num_read_kb":9687970,"num_write":2444281,"num_write_kb":9775653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20019564544,"data_stored":24331043440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80348,"ondisk_log_size":80348,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739405,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5350940,"kb_used_data":3669392,"kb_used_omap":317,"kb_used_meta":1681218,"kb_avail":89020900,"statfs":{"total":96636764160,"available":91157401600,"internally_reserved":0,"allocated":3757457408,"data_stored":4566563604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1721567894},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,4,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.69799999999999995,"15min":0.77400000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":1.9379999999999999,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.57199999999999995,"15min":0.753},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.6230000000000002,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.373}]},{"osd":1,"last update":"Sat Oct 14 01:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.1950000000000001,"15min":1.284},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17999999999999999},"max":{"1min":5.7130000000000001,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.175,"5min":1.2030000000000001,"15min":1.4730000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":5.6909999999999998,"5min":9.9329999999999998,"15min":14.586},"last":0.95899999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739390,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11168808,"kb_used_data":9777052,"kb_used_omap":822,"kb_used_meta":1390921,"kb_avail":83203032,"statfs":{"total":96636764160,"available":85199904768,"internally_reserved":0,"allocated":10011701248,"data_stored":12167107502,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":1424303362},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,2,1,1,7,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.65200000000000002,"15min":0.79000000000000004},"min":{"1min":0.23799999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.671,"5min":10.02,"15min":23.434000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.80300000000000005,"15min":0.96699999999999997},"min":{"1min":0.26400000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":1.5920000000000001,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.78800000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:22:00 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.423,"15min":1.254},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.23499999999999999},"max":{"1min":3.484,"5min":27.837,"15min":27.837},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":1.2809999999999999,"15min":1.3480000000000001},"min":{"1min":0.22700000000000001,"5min":0.20599999999999999,"15min":0.16800000000000001},"max":{"1min":3.234,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.251}]}]},{"osd":0,"up_from":8,"seq":34359739403,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6918792,"kb_used_data":6106168,"kb_used_omap":495,"kb_used_meta":812112,"kb_avail":87453048,"statfs":{"total":96636764160,"available":89551921152,"internally_reserved":0,"allocated":6252716032,"data_stored":7598149872,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":831602857},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0640000000000001,"15min":0.91000000000000003},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":5.3360000000000003,"5min":11.734999999999999,"15min":11.734999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.52800000000000002,"15min":0.58199999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.1819999999999999,"5min":3.2530000000000001,"15min":9.2439999999999998},"last":0.254}]},{"osd":2,"last update":"Sat Oct 14 01:21:46 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.82199999999999995,"15min":0.97199999999999998},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.7349999999999999,"5min":9.8059999999999992,"15min":10.539999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.71899999999999997},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":3.6789999999999998,"5min":3.7770000000000001,"15min":8.9710000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6251515904,"data_stored":7597282938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10011574272,"data_stored":12167313720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3756474368,"data_stored":4566446782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:22:44.181 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:44.181 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:44.182 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:46.268 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:46.269+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:49.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:49.437+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:49.484 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:49.485+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:49.521 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:49.521+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:49.521 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:49.521+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:49.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:49.521+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:50.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:50.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:50.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:50.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:50.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:50.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:50.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:50.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:50.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:50.589+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:50.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:50.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:51.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:51.133+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:51.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:51.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:51.181 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:51.181+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:51.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:51.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:51.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:51.997+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:52.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:52.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:52.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:52.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:52.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:52.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:52.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:52.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:52.012 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:52.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:53.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:53.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:53.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:53.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:53.606 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:53.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:53.607 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:53.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:53.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:53.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:53.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:53.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:53.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:53.657+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:53.674 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:53.673+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:22:53.676 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:53.677+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:53.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:53.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:22:53.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:53.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:22:53.702 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:53.701+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:22:54.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:54.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:22:55.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:55.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:55.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:55.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:55.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:55.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:55.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:55.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:55.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:55.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:55.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:55.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:56.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:56.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:56.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.069+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:56.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:56.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:56.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:22:56.403 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:22:56.401+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:22:56.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:56.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:56.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:56.704 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:56.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:22:58.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:58.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:22:58.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:58.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:22:58.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:22:58.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:22:58.614 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.613+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:22:58.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:58.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:22:59.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:22:59.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:23:00.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:00.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:00.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:00.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:00.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.037+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:00.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:00.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:00.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:00.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:02.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:02.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:02.525 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:02.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:02.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:02.537+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:03.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:03.073+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:03.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:03.077+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:03.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:03.081+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:03.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:03.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:04.183 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:04.615 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:04.609+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:04.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:04.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:04.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:04.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:05.033 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:05.033 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.025+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:05.062 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:05.063 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.061+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:05.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:05.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:05.069 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:05.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:05.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.181+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:05.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:05.240 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:23:05.241 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:23:05.241 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:05.241 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:05.260 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:05.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:05.282 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2759,"stamp":"2023-10-14T01:23:03.850718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29258459992,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2493161,"num_read_kb":9721199,"num_write":2452775,"num_write_kb":9811501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80481,"ondisk_log_size":80481,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23605836,"kb_used_data":19618032,"kb_used_omap":1635,"kb_used_meta":3986140,"kb_avail":259509684,"statfs":{"total":289910292480,"available":265737916416,"internally_reserved":0,"allocated":20088864768,"data_stored":24398811058,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":4081807425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,10,16,14,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22413312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5558,"num_read_kb":22184,"num_write":5467,"num_write_kb":21868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005280"},"pg_stats":[{"pgid":"2.7","version":"154'329502","reported_seq":646646,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:03.647909+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:23:03.647909+0000","last_peered":"2023-10-14T01:23:03.647909+0000","last_clean":"2023-10-14T01:23:03.647909+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:23:03.647909+0000","last_undegraded":"2023-10-14T01:23:03.647909+0000","last_fullsized":"2023-10-14T01:23:03.647909+0000","mapping_epoch":13,"log_start":"150'319402","ondisk_log_start":"150'319402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":366,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 209s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3701768630,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":317922,"num_read_kb":1234063,"num_write":310105,"num_write_kb":1240178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'329574","reported_seq":647008,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:03.626858+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:23:03.626858+0000","last_peered":"2023-10-14T01:23:03.626858+0000","last_clean":"2023-10-14T01:23:03.626858+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:23:03.626858+0000","last_undegraded":"2023-10-14T01:23:03.626858+0000","last_fullsized":"2023-10-14T01:23:03.626858+0000","mapping_epoch":13,"log_start":"150'319513","ondisk_log_start":"150'319513","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3692962160,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":319629,"num_read_kb":1230921,"num_write":309711,"num_write_kb":1238471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'298040","reported_seq":593121,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:03.613495+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:23:03.613495+0000","last_peered":"2023-10-14T01:23:03.613495+0000","last_clean":"2023-10-14T01:23:03.613495+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:23:03.613495+0000","last_undegraded":"2023-10-14T01:23:03.613495+0000","last_fullsized":"2023-10-14T01:23:03.613495+0000","mapping_epoch":13,"log_start":"150'288007","ondisk_log_start":"150'288007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":620,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 222s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3534889362,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":295386,"num_read_kb":1165488,"num_write":297072,"num_write_kb":1188258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'291012","reported_seq":582186,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:03.574779+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:23:03.574779+0000","last_peered":"2023-10-14T01:23:03.574779+0000","last_clean":"2023-10-14T01:23:03.574779+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:23:03.574779+0000","last_undegraded":"2023-10-14T01:23:03.574779+0000","last_fullsized":"2023-10-14T01:23:03.574779+0000","mapping_epoch":13,"log_start":"150'281007","ondisk_log_start":"150'281007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3457572995,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":291417,"num_read_kb":1147881,"num_write":289736,"num_write_kb":1158866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"154'328105","reported_seq":656789,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:00.970454+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:23:00.970454+0000","last_peered":"2023-10-14T01:23:00.970454+0000","last_clean":"2023-10-14T01:23:00.970454+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:23:00.970454+0000","last_undegraded":"2023-10-14T01:23:00.970454+0000","last_fullsized":"2023-10-14T01:23:00.970454+0000","mapping_epoch":13,"log_start":"150'318009","ondisk_log_start":"150'318009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3889262911,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":329003,"num_read_kb":1290062,"num_write":325607,"num_write_kb":1302281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"154'316113","reported_seq":627329,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:01.445622+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:23:01.445622+0000","last_peered":"2023-10-14T01:23:01.445622+0000","last_clean":"2023-10-14T01:23:01.445622+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:23:01.445622+0000","last_undegraded":"2023-10-14T01:23:01.445622+0000","last_fullsized":"2023-10-14T01:23:01.445622+0000","mapping_epoch":13,"log_start":"150'306109","ondisk_log_start":"150'306109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3648114957,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":311802,"num_read_kb":1210456,"num_write":306422,"num_write_kb":1225474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"154'321358","reported_seq":640129,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:01.465717+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:23:01.465717+0000","last_peered":"2023-10-14T01:23:01.465717+0000","last_clean":"2023-10-14T01:23:01.465717+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:23:01.465717+0000","last_undegraded":"2023-10-14T01:23:01.465717+0000","last_fullsized":"2023-10-14T01:23:01.465717+0000","mapping_epoch":13,"log_start":"150'311311","ondisk_log_start":"150'311311","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3700257017,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":319427,"num_read_kb":1234732,"num_write":309615,"num_write_kb":1238186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"154'315017","reported_seq":623108,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:03.597467+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:23:03.597467+0000","last_peered":"2023-10-14T01:23:03.597467+0000","last_clean":"2023-10-14T01:23:03.597467+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:23:03.597467+0000","last_undegraded":"2023-10-14T01:23:03.597467+0000","last_fullsized":"2023-10-14T01:23:03.597467+0000","mapping_epoch":13,"log_start":"150'305009","ondisk_log_start":"150'305009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3633041592,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":308449,"num_read_kb":1207487,"num_write":304338,"num_write_kb":1217239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":367,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:06.897931+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:22:06.897931+0000","last_peered":"2023-10-14T01:22:06.897931+0000","last_clean":"2023-10-14T01:22:06.897931+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:22:06.897931+0000","last_undegraded":"2023-10-14T01:22:06.897931+0000","last_fullsized":"2023-10-14T01:22:06.897931+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29257869624,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2493035,"num_read_kb":9721090,"num_write":2452606,"num_write_kb":9808953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20087382016,"data_stored":24398860912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80354,"ondisk_log_size":80354,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739409,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5430880,"kb_used_data":3681304,"kb_used_omap":317,"kb_used_meta":1749250,"kb_avail":88940960,"statfs":{"total":96636764160,"available":91075543040,"internally_reserved":0,"allocated":3769655296,"data_stored":4578761492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1791232662},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.69799999999999995,"15min":0.77400000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":1.9379999999999999,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.57199999999999995,"15min":0.753},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.6230000000000002,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.55100000000000005}]},{"osd":1,"last update":"Sat Oct 14 01:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.1950000000000001,"15min":1.284},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17999999999999999},"max":{"1min":5.7130000000000001,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.175,"5min":1.2030000000000001,"15min":1.4730000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":5.6909999999999998,"5min":9.9329999999999998,"15min":14.586},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739394,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11313088,"kb_used_data":9810164,"kb_used_omap":822,"kb_used_meta":1502089,"kb_avail":83058752,"statfs":{"total":96636764160,"available":85052162048,"internally_reserved":0,"allocated":10045607936,"data_stored":12201014190,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":1538139394},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,13,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.65200000000000002,"15min":0.79000000000000004},"min":{"1min":0.23799999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.671,"5min":10.02,"15min":23.434000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.80300000000000005,"15min":0.96699999999999997},"min":{"1min":0.26400000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":1.5920000000000001,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.23100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.4370000000000001,"15min":1.2729999999999999},"min":{"1min":0.27000000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.3310000000000004,"5min":27.837,"15min":27.837},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.3300000000000001,"15min":1.3779999999999999},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":9.1760000000000002,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739407,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6861868,"kb_used_data":6126564,"kb_used_omap":495,"kb_used_meta":734800,"kb_avail":87509972,"statfs":{"total":96636764160,"available":89610211328,"internally_reserved":0,"allocated":6273601536,"data_stored":7619035376,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":752435369},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":0.879,"15min":0.96599999999999997},"min":{"1min":0.20499999999999999,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":6.6639999999999997,"5min":6.6639999999999997,"15min":11.734999999999999},"last":2.8130000000000002},{"interface":"front","average":{"1min":1.006,"5min":0.56200000000000006,"15min":0.628},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.5110000000000001,"5min":6.5110000000000001,"15min":9.2439999999999998},"last":2.5569999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.77900000000000003,"15min":0.92100000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.472,"5min":9.8059999999999992,"15min":9.8059999999999992},"last":0.252},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.52900000000000003,"15min":0.71999999999999997},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":2.2549999999999999,"5min":3.7770000000000001,"15min":8.9710000000000001},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6272798720,"data_stored":7618565754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10045796352,"data_stored":12201535800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3768786944,"data_stored":4578759358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:23:05.284 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:05.284 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:05.285 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:06.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:06.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:06.988 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:06.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:06.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:06.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:06.989 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:06.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:07.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:07.021 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:07.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.021+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:07.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.425+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:07.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:07.486 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:07.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:07.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:07.488 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:07.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:07.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:07.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:08.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:23:08.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:08.518 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:08.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:08.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:08.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:08.550 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:08.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:08.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:08.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:10.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:10.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:10.094 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:10.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:10.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:10.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:10.097 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:10.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:10.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:10.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:10.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:10.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:10.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:10.113+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:11.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:11.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:11.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:11.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:11.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:11.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:11.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.705+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:11.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:11.751 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:11.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:11.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:11.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:11.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:11.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:11.953+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:11.992 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:11.993+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:12.015 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.013+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:12.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:12.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:12.071 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.069+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:12.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.905+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:12.927 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.929+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:12.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:12.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:12.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:12.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:13.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:13.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:23:13.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:13.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:23:14.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:14.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.057+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:14.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:14.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:14.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.089+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:14.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:14.095 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.093+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:14.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.105+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:14.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:23:14.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.129+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:23:14.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:14.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:14.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:14.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:14.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:14.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:14.157+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:15.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:15.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:15.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:15.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:15.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:15.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:15.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:15.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:15.248 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:15.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:15.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:15.248+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:15.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:15.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:15.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:15.588+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:15.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:15.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:15.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:15.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:15.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:15.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:17.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:17.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:17.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:17.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:17.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:17.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:17.501 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:17.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:17.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:17.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:18.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:18.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:18.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:18.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:18.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:18.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.732+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:18.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:18.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:18.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:20.534 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:20.532+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:20.541 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:20.542 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:20.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:20.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:20.540+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:21.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:21.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:21.660 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:21.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:21.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:21.664+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:21.685 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:21.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:22.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:22.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:22.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:22.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:22.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:22.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:23.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.400+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:23.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:23.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:23.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:23.505 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:23.517 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:23.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:23.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:23.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:23.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:25.286 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:25.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:25.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:25.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:25.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:25.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:25.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:25.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:25.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:26.102 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:26.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:26.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:26.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:26.104 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:23:26.104 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:23:26.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:26.143 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2770,"stamp":"2023-10-14T01:23:25.859973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29292477272,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2501578,"num_read_kb":9754779,"num_write":2461115,"num_write_kb":9844861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80497,"ondisk_log_size":80497,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23923900,"kb_used_data":19686176,"kb_used_omap":1635,"kb_used_meta":4236060,"kb_avail":259191620,"statfs":{"total":289910292480,"available":265412218880,"internally_reserved":0,"allocated":20158644224,"data_stored":24468590514,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":4337725505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,6,13,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22499328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5439,"num_read_kb":21700,"num_write":5513,"num_write_kb":22052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001998"},"pg_stats":[{"pgid":"2.7","version":"154'330644","reported_seq":648894,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:24.301256+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:23:24.301256+0000","last_peered":"2023-10-14T01:23:24.301256+0000","last_clean":"2023-10-14T01:23:24.301256+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:23:24.301256+0000","last_undegraded":"2023-10-14T01:23:24.301256+0000","last_fullsized":"2023-10-14T01:23:24.301256+0000","mapping_epoch":13,"log_start":"151'320602","ondisk_log_start":"151'320602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":414,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 230s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3706065334,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":319002,"num_read_kb":1238383,"num_write":311164,"num_write_kb":1244414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'330690","reported_seq":649212,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:24.284314+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:23:24.284314+0000","last_peered":"2023-10-14T01:23:24.284314+0000","last_clean":"2023-10-14T01:23:24.284314+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:23:24.284314+0000","last_undegraded":"2023-10-14T01:23:24.284314+0000","last_fullsized":"2023-10-14T01:23:24.284314+0000","mapping_epoch":13,"log_start":"151'320613","ondisk_log_start":"151'320613","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3697320304,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":320700,"num_read_kb":1235193,"num_write":310771,"num_write_kb":1242711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'299139","reported_seq":595260,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:24.255924+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:23:24.255924+0000","last_peered":"2023-10-14T01:23:24.255924+0000","last_clean":"2023-10-14T01:23:24.255924+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:23:24.255924+0000","last_undegraded":"2023-10-14T01:23:24.255924+0000","last_fullsized":"2023-10-14T01:23:24.255924+0000","mapping_epoch":13,"log_start":"150'289107","ondisk_log_start":"150'289107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":681,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 243s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3539235218,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":296416,"num_read_kb":1169592,"num_write":298137,"num_write_kb":1192518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'291994","reported_seq":584206,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:24.239726+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:23:24.239726+0000","last_peered":"2023-10-14T01:23:24.239726+0000","last_clean":"2023-10-14T01:23:24.239726+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:23:24.239726+0000","last_undegraded":"2023-10-14T01:23:24.239726+0000","last_fullsized":"2023-10-14T01:23:24.239726+0000","mapping_epoch":13,"log_start":"150'281907","ondisk_log_start":"150'281907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3461578883,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":292456,"num_read_kb":1152017,"num_write":290718,"num_write_kb":1162794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"154'329217","reported_seq":658989,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:21.256148+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:23:21.256148+0000","last_peered":"2023-10-14T01:23:21.256148+0000","last_clean":"2023-10-14T01:23:21.256148+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:23:21.256148+0000","last_undegraded":"2023-10-14T01:23:21.256148+0000","last_fullsized":"2023-10-14T01:23:21.256148+0000","mapping_epoch":13,"log_start":"151'319209","ondisk_log_start":"151'319209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3893776703,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":330085,"num_read_kb":1294386,"num_write":326713,"num_write_kb":1306705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"154'317126","reported_seq":629333,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:21.598687+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:23:21.598687+0000","last_peered":"2023-10-14T01:23:21.598687+0000","last_clean":"2023-10-14T01:23:21.598687+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:23:21.598687+0000","last_undegraded":"2023-10-14T01:23:21.598687+0000","last_fullsized":"2023-10-14T01:23:21.598687+0000","mapping_epoch":13,"log_start":"150'307109","ondisk_log_start":"150'307109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3652112653,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":312777,"num_read_kb":1214344,"num_write":307403,"num_write_kb":1229398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"154'322405","reported_seq":642276,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:21.597328+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:23:21.597328+0000","last_peered":"2023-10-14T01:23:21.597328+0000","last_clean":"2023-10-14T01:23:21.597328+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:23:21.597328+0000","last_undegraded":"2023-10-14T01:23:21.597328+0000","last_fullsized":"2023-10-14T01:23:21.597328+0000","mapping_epoch":13,"log_start":"151'312311","ondisk_log_start":"151'312311","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3704385785,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":320508,"num_read_kb":1239052,"num_write":310626,"num_write_kb":1242230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"154'316122","reported_seq":625282,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:24.202388+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:23:24.202388+0000","last_peered":"2023-10-14T01:23:24.202388+0000","last_clean":"2023-10-14T01:23:24.202388+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:23:24.202388+0000","last_undegraded":"2023-10-14T01:23:24.202388+0000","last_fullsized":"2023-10-14T01:23:24.202388+0000","mapping_epoch":13,"log_start":"150'306109","ondisk_log_start":"150'306109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3637412024,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":309508,"num_read_kb":1211703,"num_write":305414,"num_write_kb":1221543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":367,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:06.897931+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:22:06.897931+0000","last_peered":"2023-10-14T01:22:06.897931+0000","last_clean":"2023-10-14T01:22:06.897931+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:22:06.897931+0000","last_undegraded":"2023-10-14T01:22:06.897931+0000","last_fullsized":"2023-10-14T01:22:06.897931+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29291886904,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2501452,"num_read_kb":9754670,"num_write":2460946,"num_write_kb":9842313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20155826176,"data_stored":24467305072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80370,"ondisk_log_size":80370,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739413,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5456988,"kb_used_data":3693524,"kb_used_omap":317,"kb_used_meta":1763138,"kb_avail":88914852,"statfs":{"total":96636764160,"available":91048808448,"internally_reserved":0,"allocated":3782168576,"data_stored":4591274772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1805453974},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.69799999999999995,"15min":0.77400000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":1.9379999999999999,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.57199999999999995,"15min":0.753},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.19800000000000001},"max":{"1min":5.6230000000000002,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.20999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:22:34 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.1950000000000001,"15min":1.284},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17999999999999999},"max":{"1min":5.7130000000000001,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.2030000000000001,"15min":1.4730000000000001},"min":{"1min":0.27100000000000002,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":5.6909999999999998,"5min":9.9329999999999998,"15min":14.586},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359739398,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11530168,"kb_used_data":9845360,"kb_used_omap":822,"kb_used_meta":1683977,"kb_avail":82841672,"statfs":{"total":96636764160,"available":84829872128,"internally_reserved":0,"allocated":10081648640,"data_stored":12237054894,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":1724392706},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,8,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.65200000000000002,"15min":0.79000000000000004},"min":{"1min":0.23799999999999999,"5min":0.183,"15min":0.183},"max":{"1min":1.671,"5min":10.02,"15min":23.434000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.80300000000000005,"15min":0.96699999999999997},"min":{"1min":0.26400000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":1.5920000000000001,"5min":10.074999999999999,"15min":24.908000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.4370000000000001,"15min":1.2729999999999999},"min":{"1min":0.27000000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.3310000000000004,"5min":27.837,"15min":27.837},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.3300000000000001,"15min":1.3779999999999999},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":9.1760000000000002,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.41099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739411,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6936744,"kb_used_data":6147292,"kb_used_omap":495,"kb_used_meta":788944,"kb_avail":87435096,"statfs":{"total":96636764160,"available":89533538304,"internally_reserved":0,"allocated":6294827008,"data_stored":7640260848,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":807878825},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":0.879,"15min":0.96599999999999997},"min":{"1min":0.20499999999999999,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":6.6639999999999997,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":1.006,"5min":0.56200000000000006,"15min":0.628},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.5110000000000001,"5min":6.5110000000000001,"15min":9.2439999999999998},"last":0.38600000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.77900000000000003,"15min":0.92100000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.472,"5min":9.8059999999999992,"15min":9.8059999999999992},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.52900000000000003,"15min":0.71999999999999997},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":2.2549999999999999,"5min":3.7770000000000001,"15min":8.9710000000000001},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6294171648,"data_stored":7639938682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10080337920,"data_stored":12236077368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3781316608,"data_stored":4591289022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:23:26.145 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:26.145 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:26.146 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:26.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:26.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:26.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:26.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:26.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.280+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:26.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:26.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:26.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.316+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:26.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.532+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:26.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:26.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:26.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.548+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:26.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:26.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:26.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:26.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:27.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:27.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:27.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:27.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:27.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:27.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:30.202 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:30.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:30.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:30.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:30.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:30.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:31.198 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.196+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:31.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:31.204 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:31.233 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:31.233 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:31.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.232+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:31.797 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.796+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:31.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:31.857 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.852+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:31.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:31.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:31.881 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:31.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:34.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:34.035 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:34.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:34.064 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:34.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.084+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:34.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:34.909 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:34.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:34.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:34.952 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.948+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:34.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:34.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:34.982 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:34.980+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:36.760 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:36.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:36.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:36.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:36.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:36.816+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:36.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:36.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:37.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:37.792+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:37.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:37.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:37.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:37.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:37.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:37.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:37.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:37.808+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:37.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:37.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:37.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:37.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:38.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.744+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:38.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.744+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:38.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:38.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:38.771 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:38.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:38.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:38.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:38.868 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:38.879 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:38.893 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:38.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:43.252 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:43.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:43.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:43.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:43.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:43.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:43.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:43.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:43.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:43.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:43.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:43.676+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:43.678 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:43.676+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:43.684 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:43.684+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:43.711 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:43.708+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:43.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:43.724+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:43.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:43.724+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:43.728 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:43.724+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:45.060 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.060+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:45.072 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.072+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:45.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:45.086 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:45.087 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.084+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:45.114 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:45.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:45.115 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:45.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:45.116 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:45.112+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:45.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:45.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:45.133 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:45.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:45.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:45.144+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:45.639 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:45.640 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.636+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:45.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.644+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:45.651 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.648+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:45.674 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.672+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:45.675 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.672+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:45.675 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:45.672+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:46.147 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:46.576 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:23:46.576+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:23:46.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:46.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:46.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:46.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:46.702 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:46.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:46.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:46.724+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:47.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:47.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:47.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:47.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:47.076 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:23:47.077 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:23:47.077 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:47.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:47.092 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:47.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:47.099 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:47.100+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:47.120 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:47.116+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:47.171 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2780,"stamp":"2023-10-14T01:23:45.866702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29326560088,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2509844,"num_read_kb":9787763,"num_write":2469457,"num_write_kb":9878229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80581,"ondisk_log_size":80581,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24038116,"kb_used_data":19752972,"kb_used_omap":1635,"kb_used_meta":4283484,"kb_avail":259077404,"statfs":{"total":289910292480,"available":265295261696,"internally_reserved":0,"allocated":20227043328,"data_stored":24536989618,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":4386287681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,9,20,11,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22253568,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5434,"num_read_kb":21684,"num_write":5445,"num_write_kb":21780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005432"},"pg_stats":[{"pgid":"2.7","version":"154'331812","reported_seq":651127,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:44.809295+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:23:44.809295+0000","last_peered":"2023-10-14T01:23:44.809295+0000","last_clean":"2023-10-14T01:23:44.809295+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:23:44.809295+0000","last_undegraded":"2023-10-14T01:23:44.809295+0000","last_fullsized":"2023-10-14T01:23:44.809295+0000","mapping_epoch":13,"log_start":"152'321802","ondisk_log_start":"152'321802","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":458,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 250s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3710513590,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":320044,"num_read_kb":1242547,"num_write":312253,"num_write_kb":1248770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"154'331809","reported_seq":651450,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:44.820155+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:23:44.820155+0000","last_peered":"2023-10-14T01:23:44.820155+0000","last_clean":"2023-10-14T01:23:44.820155+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:23:44.820155+0000","last_undegraded":"2023-10-14T01:23:44.820155+0000","last_fullsized":"2023-10-14T01:23:44.820155+0000","mapping_epoch":13,"log_start":"152'321713","ondisk_log_start":"152'321713","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3701674352,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":321803,"num_read_kb":1239589,"num_write":311834,"num_write_kb":1246963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"154'300180","reported_seq":597282,"reported_epoch":154,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:44.811364+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:23:44.811364+0000","last_peered":"2023-10-14T01:23:44.811364+0000","last_clean":"2023-10-14T01:23:44.811364+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:23:44.811364+0000","last_undegraded":"2023-10-14T01:23:44.811364+0000","last_fullsized":"2023-10-14T01:23:44.811364+0000","mapping_epoch":13,"log_start":"151'290107","ondisk_log_start":"151'290107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":766,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 263s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3543298450,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":297385,"num_read_kb":1173468,"num_write":299130,"num_write_kb":1196490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"154'292989","reported_seq":586227,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:44.779002+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:23:44.779002+0000","last_peered":"2023-10-14T01:23:44.779002+0000","last_clean":"2023-10-14T01:23:44.779002+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:23:44.779002+0000","last_undegraded":"2023-10-14T01:23:44.779002+0000","last_fullsized":"2023-10-14T01:23:44.779002+0000","mapping_epoch":13,"log_start":"150'282907","ondisk_log_start":"150'282907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3465629827,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":293484,"num_read_kb":1156125,"num_write":291713,"num_write_kb":1166774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"154'330352","reported_seq":661271,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:41.513220+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:23:41.513220+0000","last_peered":"2023-10-14T01:23:41.513220+0000","last_clean":"2023-10-14T01:23:41.513220+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:23:41.513220+0000","last_undegraded":"2023-10-14T01:23:41.513220+0000","last_fullsized":"2023-10-14T01:23:41.513220+0000","mapping_epoch":13,"log_start":"152'320309","ondisk_log_start":"152'320309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3898396991,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":331230,"num_read_kb":1298950,"num_write":327844,"num_write_kb":1311229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"154'318146","reported_seq":631374,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:41.715874+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:23:41.715874+0000","last_peered":"2023-10-14T01:23:41.715874+0000","last_clean":"2023-10-14T01:23:41.715874+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:23:41.715874+0000","last_undegraded":"2023-10-14T01:23:41.715874+0000","last_fullsized":"2023-10-14T01:23:41.715874+0000","mapping_epoch":13,"log_start":"151'308109","ondisk_log_start":"151'308109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3656220941,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":313782,"num_read_kb":1218356,"num_write":308411,"num_write_kb":1233430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"154'323451","reported_seq":644327,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:41.703591+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:23:41.703591+0000","last_peered":"2023-10-14T01:23:41.703591+0000","last_clean":"2023-10-14T01:23:41.703591+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:23:41.703591+0000","last_undegraded":"2023-10-14T01:23:41.703591+0000","last_fullsized":"2023-10-14T01:23:41.703591+0000","mapping_epoch":13,"log_start":"152'313411","ondisk_log_start":"152'313411","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3708604665,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":321492,"num_read_kb":1242968,"num_write":311657,"num_write_kb":1246354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"154'317182","reported_seq":627339,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:23:44.786873+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:23:44.786873+0000","last_peered":"2023-10-14T01:23:44.786873+0000","last_clean":"2023-10-14T01:23:44.786873+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:23:44.786873+0000","last_undegraded":"2023-10-14T01:23:44.786873+0000","last_fullsized":"2023-10-14T01:23:44.786873+0000","mapping_epoch":13,"log_start":"151'307109","ondisk_log_start":"151'307109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3641630904,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":310498,"num_read_kb":1215651,"num_write":306446,"num_write_kb":1225671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":367,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T01:22:06.897931+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:22:06.897931+0000","last_peered":"2023-10-14T01:22:06.897931+0000","last_clean":"2023-10-14T01:22:06.897931+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:22:06.897931+0000","last_undegraded":"2023-10-14T01:22:06.897931+0000","last_fullsized":"2023-10-14T01:22:06.897931+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29325969720,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2509718,"num_read_kb":9787654,"num_write":2469288,"num_write_kb":9875681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20224073728,"data_stored":24535552624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80454,"ondisk_log_size":80454,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739417,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5519084,"kb_used_data":3705636,"kb_used_omap":317,"kb_used_meta":1813122,"kb_avail":88852756,"statfs":{"total":96636764160,"available":90985222144,"internally_reserved":0,"allocated":3794571264,"data_stored":4603677460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1856637590},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":2.7919999999999998,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.70999999999999996},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":3.9060000000000001,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:23:35 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.3480000000000001,"15min":1.351},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":8.6159999999999997,"5min":14.005000000000001,"15min":14.005000000000001},"last":2.7240000000000002},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.248,"15min":1.4710000000000001},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":4.3600000000000003,"5min":9.9329999999999998,"15min":14.586},"last":0.29499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739402,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11674268,"kb_used_data":9878356,"kb_used_omap":822,"kb_used_meta":1795081,"kb_avail":82697572,"statfs":{"total":96636764160,"available":84682313728,"internally_reserved":0,"allocated":10115436544,"data_stored":12270842798,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":1838163202},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,8,11,7,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":0.73699999999999999,"15min":0.85999999999999999},"min":{"1min":0.29899999999999999,"5min":0.183,"15min":0.183},"max":{"1min":6.1079999999999997,"5min":6.1079999999999997,"15min":23.434000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.895,"5min":0.93700000000000006,"15min":1.0640000000000001},"min":{"1min":0.23100000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":8.6460000000000008,"5min":8.6460000000000008,"15min":24.908000000000001},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Oct 14 01:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.4370000000000001,"15min":1.2729999999999999},"min":{"1min":0.27000000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.3310000000000004,"5min":27.837,"15min":27.837},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.3300000000000001,"15min":1.3779999999999999},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":9.1760000000000002,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739415,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6844764,"kb_used_data":6168980,"kb_used_omap":495,"kb_used_meta":675280,"kb_avail":87527076,"statfs":{"total":96636764160,"available":89627725824,"internally_reserved":0,"allocated":6317035520,"data_stored":7662469360,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":691486889},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:46 2023","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":0.879,"15min":0.96599999999999997},"min":{"1min":0.20499999999999999,"5min":0.20300000000000001,"15min":0.17599999999999999},"max":{"1min":6.6639999999999997,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.006,"5min":0.56200000000000006,"15min":0.628},"min":{"1min":0.21099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":6.5110000000000001,"5min":6.5110000000000001,"15min":9.2439999999999998},"last":0.23100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.77900000000000003,"15min":0.92100000000000004},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.472,"5min":9.8059999999999992,"15min":9.8059999999999992},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.52900000000000003,"15min":0.71999999999999997},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":2.2549999999999999,"5min":3.7770000000000001,"15min":8.9710000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6315614208,"data_stored":7661381242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10114801664,"data_stored":12270541112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3793657856,"data_stored":4603630270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:23:47.172 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:47.173 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:47.173 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:47.996 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:47.996+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:48.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:48.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:48.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.012+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:48.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:48.024 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.024+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:48.047 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:48.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:48.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.540+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:48.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.556+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:48.592 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:48.603 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.600+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:48.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:48.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:48.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:48.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:48.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:49.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:49.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:49.772 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:49.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:49.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:49.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:49.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:49.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:50.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:50.772+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:50.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:50.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:50.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:50.780+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:50.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:50.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:51.076 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:23:51.076+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:23:51.105 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:51.104+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:51.128 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:51.128+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:51.142 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:51.140+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:51.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:51.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:51.173 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:51.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:51.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:51.172+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:51.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:51.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:51.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:51.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:51.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:51.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:51.300 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:51.320 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:51.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:51.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:52.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:52.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:52.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:52.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.700+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:52.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:52.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:52.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:52.853 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:52.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:52.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:52.876+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:53.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:53.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:53.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:53.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:53.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:53.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:53.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:53.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:53.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:53.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:54.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:54.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.268+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:54.312 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.304+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:54.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:54.331 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:54.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.328+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:54.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:54.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:54.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:54.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:55.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:55.916 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.916+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:55.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:55.951 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:55.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:55.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:55.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.948+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:55.989 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:55.988+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:56.004 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.000+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:56.022 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:56.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:56.023 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:56.880 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.880+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:56.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:56.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:56.941 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:56.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:56.942 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:56.936+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:57.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:57.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:23:57.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:57.504+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:23:57.517 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:57.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:23:57.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:57.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:23:57.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:57.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:23:57.518 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:57.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:23:57.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:57.516+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:23:58.289 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:58.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:58.519 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.516+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:23:58.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:58.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:23:58.575 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:23:58.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:23:58.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.584+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:58.598 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:58.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:23:58.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:23:58.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:23:58.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.656+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:23:58.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:23:58.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:00.001 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:59.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:24:00.002 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:23:59.996+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:00.011 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:00.012+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:00.019 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:00.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:00.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:00.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:00.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:24:00.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:24:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:00.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:00.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.332+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:00.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.340+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:00.383 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.380+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:00.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:00.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:24:00.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.404+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:24:00.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:00.414 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.412+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:00.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:00.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:00.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:00.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:00.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:00.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:00.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:01.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:01.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:01.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:01.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:01.275 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:01.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.300+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:01.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:01.319 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.320+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:01.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:01.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:01.622 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.620+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:01.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:01.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:06.250 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.248+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:06.257 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.256+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:06.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:06.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:06.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:06.315 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:06.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:06.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:06.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:06.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:06.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:06.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:06.954 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:06.952+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:06.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:06.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:06.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.956+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:06.966 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:06.964+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:06.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:06.972+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:06.982 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:06.980+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:07.175 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:07.259 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:07.256+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:07.350 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:07.348+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:07.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:07.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:07.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:07.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:07.369 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:07.368+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:07.982 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:24:07.982 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:24:08.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:08.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:08.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:08.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:08.045 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:08.036+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:08.058 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:08.052+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:08.064 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2792,"stamp":"2023-10-14T01:24:07.874156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29363874648,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2518862,"num_read_kb":9823755,"num_write":2478596,"num_write_kb":9914785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80491,"ondisk_log_size":80491,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24433916,"kb_used_data":19828124,"kb_used_omap":1635,"kb_used_meta":4604124,"kb_avail":258681604,"statfs":{"total":289910292480,"available":264889962496,"internally_reserved":0,"allocated":20303998976,"data_stored":24613932537,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":4714623041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,10,10,10,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20455424,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4906,"num_read_kb":19572,"num_write":5005,"num_write_kb":20020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001882"},"pg_stats":[{"pgid":"2.7","version":"155'333028","reported_seq":653446,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:05.542132+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:24:05.542132+0000","last_peered":"2023-10-14T01:24:05.542132+0000","last_clean":"2023-10-14T01:24:05.542132+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:24:05.542132+0000","last_undegraded":"2023-10-14T01:24:05.542132+0000","last_fullsized":"2023-10-14T01:24:05.542132+0000","mapping_epoch":13,"log_start":"152'323002","ondisk_log_start":"152'323002","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":488,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 271s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3715097014,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":321124,"num_read_kb":1246855,"num_write":313374,"num_write_kb":1253254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'332922","reported_seq":653620,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.535923+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:24:05.535923+0000","last_peered":"2023-10-14T01:24:05.535923+0000","last_clean":"2023-10-14T01:24:05.535923+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:24:05.535923+0000","last_undegraded":"2023-10-14T01:24:05.535923+0000","last_fullsized":"2023-10-14T01:24:05.535923+0000","mapping_epoch":13,"log_start":"152'322913","ondisk_log_start":"152'322913","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3705876848,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":322843,"num_read_kb":1243737,"num_write":312863,"num_write_kb":1251079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'301244","reported_seq":599368,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:05.489573+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:24:05.489573+0000","last_peered":"2023-10-14T01:24:05.489573+0000","last_clean":"2023-10-14T01:24:05.489573+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:24:05.489573+0000","last_undegraded":"2023-10-14T01:24:05.489573+0000","last_fullsized":"2023-10-14T01:24:05.489573+0000","mapping_epoch":13,"log_start":"152'291207","ondisk_log_start":"152'291207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":813,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 284s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3547521426,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":298399,"num_read_kb":1177512,"num_write":300170,"num_write_kb":1200650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'293948","reported_seq":588126,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.484097+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:24:05.484097+0000","last_peered":"2023-10-14T01:24:05.484097+0000","last_clean":"2023-10-14T01:24:05.484097+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:24:05.484097+0000","last_undegraded":"2023-10-14T01:24:05.484097+0000","last_fullsized":"2023-10-14T01:24:05.484097+0000","mapping_epoch":13,"log_start":"152'283907","ondisk_log_start":"152'283907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3469553795,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":294421,"num_read_kb":1159873,"num_write":292672,"num_write_kb":1170610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"155'331806","reported_seq":664111,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.691726+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:24:06.691726+0000","last_peered":"2023-10-14T01:24:06.691726+0000","last_clean":"2023-10-14T01:24:06.691726+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:24:06.691726+0000","last_undegraded":"2023-10-14T01:24:06.691726+0000","last_fullsized":"2023-10-14T01:24:06.691726+0000","mapping_epoch":13,"log_start":"152'321709","ondisk_log_start":"152'321709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3904307519,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":332608,"num_read_kb":1304454,"num_write":329290,"num_write_kb":1317013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"155'319468","reported_seq":633982,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.837674+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:24:06.837674+0000","last_peered":"2023-10-14T01:24:06.837674+0000","last_clean":"2023-10-14T01:24:06.837674+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:24:06.837674+0000","last_undegraded":"2023-10-14T01:24:06.837674+0000","last_fullsized":"2023-10-14T01:24:06.837674+0000","mapping_epoch":13,"log_start":"152'309409","ondisk_log_start":"152'309409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3661402381,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":315047,"num_read_kb":1223408,"num_write":309681,"num_write_kb":1238510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"155'324779","reported_seq":646943,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.826427+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:24:06.826427+0000","last_peered":"2023-10-14T01:24:06.826427+0000","last_clean":"2023-10-14T01:24:06.826427+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:24:06.826427+0000","last_undegraded":"2023-10-14T01:24:06.826427+0000","last_fullsized":"2023-10-14T01:24:06.826427+0000","mapping_epoch":13,"log_start":"152'314711","ondisk_log_start":"152'314711","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3713769721,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":322757,"num_read_kb":1248020,"num_write":312920,"num_write_kb":1251406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"155'318236","reported_seq":629442,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.491308+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:24:05.491308+0000","last_peered":"2023-10-14T01:24:05.491308+0000","last_clean":"2023-10-14T01:24:05.491308+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:24:05.491308+0000","last_undegraded":"2023-10-14T01:24:05.491308+0000","last_fullsized":"2023-10-14T01:24:05.491308+0000","mapping_epoch":13,"log_start":"152'308209","ondisk_log_start":"152'308209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3645755576,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":311537,"num_read_kb":1219787,"num_write":307457,"num_write_kb":1229715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":368,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T01:22:39.852230+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:22:39.852230+0000","last_peered":"2023-10-14T01:22:39.852230+0000","last_clean":"2023-10-14T01:22:39.852230+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:22:39.852230+0000","last_undegraded":"2023-10-14T01:22:39.852230+0000","last_fullsized":"2023-10-14T01:22:39.852230+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29363284280,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2518736,"num_read_kb":9823646,"num_write":2478427,"num_write_kb":9912237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20300664832,"data_stored":24612143728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80364,"ondisk_log_size":80364,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739422,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5602096,"kb_used_data":3720680,"kb_used_omap":317,"kb_used_meta":1881090,"kb_avail":88769744,"statfs":{"total":96636764160,"available":90900217856,"internally_reserved":0,"allocated":3809976320,"data_stored":4619078273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1926236822},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":2.7919999999999998,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.70999999999999996},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":3.9060000000000001,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.312}]},{"osd":1,"last update":"Sat Oct 14 01:23:35 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.3480000000000001,"15min":1.351},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":8.6159999999999997,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.248,"15min":1.4710000000000001},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":4.3600000000000003,"5min":9.9329999999999998,"15min":14.586},"last":0.42699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739406,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11861312,"kb_used_data":9912688,"kb_used_omap":822,"kb_used_meta":1947785,"kb_avail":82510528,"statfs":{"total":96636764160,"available":84490780672,"internally_reserved":0,"allocated":10150592512,"data_stored":12305994523,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":1994532098},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,9,8,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":0.73699999999999999,"15min":0.85999999999999999},"min":{"1min":0.29899999999999999,"5min":0.183,"15min":0.183},"max":{"1min":6.1079999999999997,"5min":6.1079999999999997,"15min":23.434000000000001},"last":0.22},{"interface":"front","average":{"1min":1.895,"5min":0.93700000000000006,"15min":1.0640000000000001},"min":{"1min":0.23100000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":8.6460000000000008,"5min":8.6460000000000008,"15min":24.908000000000001},"last":0.251}]},{"osd":2,"last update":"Sat Oct 14 01:24:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4350000000000001,"15min":1.3140000000000001},"min":{"1min":0.26400000000000001,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.2530000000000001,"5min":27.837,"15min":27.837},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.591,"15min":1.454},"min":{"1min":0.246,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":10.411,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.35499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739420,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6970508,"kb_used_data":6194756,"kb_used_omap":495,"kb_used_meta":775248,"kb_avail":87401332,"statfs":{"total":96636764160,"available":89498963968,"internally_reserved":0,"allocated":6343430144,"data_stored":7688859741,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":793854121},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.96299999999999997},"min":{"1min":0.185,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":3.6309999999999998,"5min":6.6639999999999997,"15min":11.734999999999999},"last":2.2170000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.64100000000000001,"15min":0.61199999999999999},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.7619999999999996,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.70899999999999996,"15min":0.89200000000000002},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.1890000000000001,"5min":7.1970000000000001,"15min":9.8059999999999992},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.50600000000000001,"15min":0.71199999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":1.7070000000000001,"5min":3.6789999999999998,"15min":8.9710000000000001},"last":0.251}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6342242304,"data_stored":7688009338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10149289984,"data_stored":12305029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3809132544,"data_stored":4619104958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:24:08.066 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:08.066 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:08.067 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:24:08.819 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:24:08.856 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:24:09.695 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:24:09.753 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:09.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:09.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:09.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:09.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:09.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:09.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:09.910 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:09.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:09.935 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:09.931+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:09.940 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:09.939+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:09.949 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:09.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:09.949 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:09.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:09.950 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:09.947+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:13.316 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:13.311+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:13.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:13.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:13.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:13.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:13.317 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:13.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:13.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:13.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:13.355 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:13.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:14.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:14.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:14.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:14.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:14.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:14.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:14.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:14.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:14.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:14.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:15.770 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:15.767+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:15.776 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:15.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:15.776 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:15.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:15.776 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:15.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:15.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:15.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:15.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:15.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:24:15.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:15.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:24:15.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:15.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:15.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:15.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:15.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:15.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:15.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:15.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:15.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:15.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:16.006 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:16.003+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:16.020 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:16.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:16.059 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:16.055+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:16.946 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:16.943+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:16.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:16.959+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:18.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:18.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:18.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:18.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:18.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:18.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:18.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:18.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:18.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:18.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:19.100 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:19.099+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:19.126 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:19.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:19.126 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:19.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:19.127 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:19.119+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:20.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:20.563+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:20.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:20.571+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:20.587 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:20.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:20.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:20.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:20.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:20.619+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:21.835 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:21.831+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:21.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:21.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:21.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:21.895+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:21.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:21.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:21.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:21.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:21.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:21.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:21.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:21.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:21.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:21.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:21.911 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:21.907+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:21.913 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:21.911+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:22.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:22.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:24:22.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:22.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:22.961 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:22.959+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:22.971 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:22.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:22.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:22.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:22.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:22.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:22.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:22.979+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:23.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:23.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:23.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:23.527+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:23.585 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:23.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:23.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:23.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:23.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:23.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:24.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:24.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:24.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:24.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:24.781 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:24.779+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:24.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:24.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:24.811 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:24.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:25.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:25.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:25.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:25.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:25.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:25.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:25.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:25.735+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:25.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:25.739+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:26.275 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.271+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:26.293 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:26.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:26.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.291+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:26.700 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.699+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:24:26.701 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.699+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:26.724 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:26.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.727+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:26.747 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:26.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:26.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:26.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:28.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:28.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:28.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:28.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:28.809 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:28.803+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:28.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:28.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:28.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:28.891+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:28.901 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:28.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:29.073 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:29.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:29.083 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:29.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:29.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:29.131+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:29.754 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:30.214 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:24:30.215 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:24:30.243 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2803,"stamp":"2023-10-14T01:24:29.879619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29397875544,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2527075,"num_read_kb":9856515,"num_write":2486920,"num_write_kb":9948081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80614,"ondisk_log_size":80614,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24725124,"kb_used_data":19893416,"kb_used_omap":1635,"kb_used_meta":4830044,"kb_avail":258390396,"statfs":{"total":289910292480,"available":264591765504,"internally_reserved":0,"allocated":20370857984,"data_stored":24680791545,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":4945965121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,6,14,11,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17203200,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4063,"num_read_kb":16204,"num_write":4216,"num_write_kb":16864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"2.7","version":"155'334119","reported_seq":655645,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:26.395352+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:24:26.395352+0000","last_peered":"2023-10-14T01:24:26.395352+0000","last_clean":"2023-10-14T01:24:26.395352+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:24:26.395352+0000","last_undegraded":"2023-10-14T01:24:26.395352+0000","last_fullsized":"2023-10-14T01:24:26.395352+0000","mapping_epoch":13,"log_start":"152'324102","ondisk_log_start":"152'324102","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":545,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 292s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3719201206,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":322199,"num_read_kb":1251139,"num_write":314376,"num_write_kb":1257262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'334039","reported_seq":655789,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.369991+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:24:26.369991+0000","last_peered":"2023-10-14T01:24:26.369991+0000","last_clean":"2023-10-14T01:24:26.369991+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:24:26.369991+0000","last_undegraded":"2023-10-14T01:24:26.369991+0000","last_fullsized":"2023-10-14T01:24:26.369991+0000","mapping_epoch":13,"log_start":"152'324013","ondisk_log_start":"152'324013","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3710177648,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":323873,"num_read_kb":1247845,"num_write":313919,"num_write_kb":1255303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'302296","reported_seq":601420,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:26.391448+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:24:26.391448+0000","last_peered":"2023-10-14T01:24:26.391448+0000","last_clean":"2023-10-14T01:24:26.391448+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:24:26.391448+0000","last_undegraded":"2023-10-14T01:24:26.391448+0000","last_fullsized":"2023-10-14T01:24:26.391448+0000","mapping_epoch":13,"log_start":"152'292208","ondisk_log_start":"152'292208","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":855,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 305s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3551723922,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":299386,"num_read_kb":1181452,"num_write":301201,"num_write_kb":1204774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'294956","reported_seq":590096,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.374153+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:24:26.374153+0000","last_peered":"2023-10-14T01:24:26.374153+0000","last_clean":"2023-10-14T01:24:26.374153+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:24:26.374153+0000","last_undegraded":"2023-10-14T01:24:26.374153+0000","last_fullsized":"2023-10-14T01:24:26.374153+0000","mapping_epoch":13,"log_start":"152'284907","ondisk_log_start":"152'284907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3473662083,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":295381,"num_read_kb":1163705,"num_write":293680,"num_write_kb":1174642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"155'332978","reported_seq":666420,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.881219+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:24:26.881219+0000","last_peered":"2023-10-14T01:24:26.881219+0000","last_clean":"2023-10-14T01:24:26.881219+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:24:26.881219+0000","last_undegraded":"2023-10-14T01:24:26.881219+0000","last_fullsized":"2023-10-14T01:24:26.881219+0000","mapping_epoch":13,"log_start":"152'322909","ondisk_log_start":"152'322909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3909050687,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":333748,"num_read_kb":1308998,"num_write":330457,"num_write_kb":1321681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"155'320507","reported_seq":635981,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:27.080628+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:24:27.080628+0000","last_peered":"2023-10-14T01:24:27.080628+0000","last_clean":"2023-10-14T01:24:27.080628+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:24:27.080628+0000","last_undegraded":"2023-10-14T01:24:27.080628+0000","last_fullsized":"2023-10-14T01:24:27.080628+0000","mapping_epoch":13,"log_start":"152'310409","ondisk_log_start":"152'310409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3665518861,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":315991,"num_read_kb":1227168,"num_write":310680,"num_write_kb":1242506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"155'325806","reported_seq":649027,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:27.087508+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:24:27.087508+0000","last_peered":"2023-10-14T01:24:27.087508+0000","last_clean":"2023-10-14T01:24:27.087508+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:24:27.087508+0000","last_undegraded":"2023-10-14T01:24:27.087508+0000","last_fullsized":"2023-10-14T01:24:27.087508+0000","mapping_epoch":13,"log_start":"152'315711","ondisk_log_start":"152'315711","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3717734649,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":323797,"num_read_kb":1252176,"num_write":313897,"num_write_kb":1255314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"155'319354","reported_seq":631607,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.369886+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:24:26.369886+0000","last_peered":"2023-10-14T01:24:26.369886+0000","last_clean":"2023-10-14T01:24:26.369886+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:24:26.369886+0000","last_undegraded":"2023-10-14T01:24:26.369886+0000","last_fullsized":"2023-10-14T01:24:26.369886+0000","mapping_epoch":13,"log_start":"152'309309","ondisk_log_start":"152'309309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3650216120,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":312574,"num_read_kb":1223923,"num_write":308541,"num_write_kb":1234051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":369,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:23:49.876368+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:23:49.876368+0000","last_peered":"2023-10-14T01:23:49.876368+0000","last_clean":"2023-10-14T01:23:49.876368+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:23:49.876368+0000","last_undegraded":"2023-10-14T01:23:49.876368+0000","last_fullsized":"2023-10-14T01:23:49.876368+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29397285176,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2526949,"num_read_kb":9856406,"num_write":2486751,"num_write_kb":9945533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20368445440,"data_stored":24679924336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80487,"ondisk_log_size":80487,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739426,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5667536,"kb_used_data":3733384,"kb_used_omap":317,"kb_used_meta":1933826,"kb_avail":88704304,"statfs":{"total":96636764160,"available":90833207296,"internally_reserved":0,"allocated":3822985216,"data_stored":4632087169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1980238486},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":2.7919999999999998,"5min":7.4669999999999996,"15min":7.6440000000000001},"last":12.356},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57999999999999996,"15min":0.70999999999999996},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":3.9060000000000001,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:23:35 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.3480000000000001,"15min":1.351},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":8.6159999999999997,"5min":14.005000000000001,"15min":14.005000000000001},"last":7.8200000000000003},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.248,"15min":1.4710000000000001},"min":{"1min":0.245,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":4.3600000000000003,"5min":9.9329999999999998,"15min":14.586},"last":7.8550000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739410,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12036368,"kb_used_data":9944968,"kb_used_omap":822,"kb_used_meta":2090569,"kb_avail":82335472,"statfs":{"total":96636764160,"available":84311523328,"internally_reserved":0,"allocated":10183647232,"data_stored":12339049243,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":2140742914},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,11,8,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":0.73699999999999999,"15min":0.85999999999999999},"min":{"1min":0.29899999999999999,"5min":0.183,"15min":0.183},"max":{"1min":6.1079999999999997,"5min":6.1079999999999997,"15min":23.434000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.895,"5min":0.93700000000000006,"15min":1.0640000000000001},"min":{"1min":0.23100000000000001,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":8.6460000000000008,"5min":8.6460000000000008,"15min":24.908000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Oct 14 01:24:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4350000000000001,"15min":1.3140000000000001},"min":{"1min":0.26400000000000001,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.2530000000000001,"5min":27.837,"15min":27.837},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.591,"15min":1.454},"min":{"1min":0.246,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":10.411,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.50600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739424,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7021220,"kb_used_data":6215064,"kb_used_omap":495,"kb_used_meta":805648,"kb_avail":87350620,"statfs":{"total":96636764160,"available":89447034880,"internally_reserved":0,"allocated":6364225536,"data_stored":7709655133,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":824983721},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.96299999999999997},"min":{"1min":0.185,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":3.6309999999999998,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.64100000000000001,"15min":0.61199999999999999},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.7619999999999996,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.70899999999999996,"15min":0.89200000000000002},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.1890000000000001,"5min":7.1970000000000001,"15min":9.8059999999999992},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.50600000000000001,"15min":0.71199999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":1.7070000000000001,"5min":3.6789999999999998,"15min":8.9710000000000001},"last":0.36599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6363185152,"data_stored":7708952186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10183729152,"data_stored":12339468600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3821531136,"data_stored":4631503550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:24:30.244 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:30.245 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:30.245 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:30.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:30.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:30.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:30.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:30.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:30.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:30.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:30.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:30.997 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:30.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:30.998 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:30.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:30.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:31.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:31.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:31.054 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:31.051+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:32.113 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:32.107+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:32.114 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:32.107+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:32.116 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:32.115+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:32.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:32.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:32.143 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:32.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:32.143 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:32.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:32.144 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:32.139+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:32.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:32.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:32.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:32.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:32.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:32.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:32.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:32.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:32.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:32.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:32.206 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:32.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:33.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:33.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:34.017 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:34.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:34.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:34.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:34.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:34.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:35.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:35.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:35.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:35.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:35.851 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:35.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:35.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:35.859+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:35.877 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:35.867+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:35.878 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:35.875+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:36.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:36.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:24:36.104 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:36.103+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:36.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:36.135+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:36.153 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:36.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:36.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:36.175+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:36.179 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:36.175+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:36.180 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:36.175+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:38.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:38.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:38.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:38.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:38.659 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:38.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:38.668 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:38.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:38.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:38.679+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:38.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:38.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:39.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:39.153 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:39.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:39.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:39.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:39.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:39.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:39.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:39.207+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:42.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:42.275+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:42.371 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:42.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:42.448 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:42.449 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.447+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:42.479 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:42.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.491+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:42.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.491+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:42.493 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.491+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:42.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.603+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:42.619 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:42.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:42.623 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:42.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:43.169 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:43.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:43.183 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:43.179+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:44.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:44.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:44.406 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:44.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:44.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:44.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:44.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:44.403+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:44.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:44.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:44.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:44.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:45.658 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:45.655+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:45.670 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:45.667+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:45.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.771+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:45.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:45.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:45.788 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:45.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.787+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:45.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.799+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:45.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:45.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:45.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:46.234 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:46.231+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:46.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:46.235+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:46.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:46.255+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:46.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:46.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:46.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:46.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:46.288 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:46.287+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:46.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:46.879+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:24:46.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:46.879+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:46.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:46.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:46.903 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:46.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:46.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:46.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:46.904 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:46.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:46.905 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:46.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:47.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:47.823+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:47.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:47.843+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:47.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:47.855+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:48.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:48.276 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.271+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:48.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:48.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:48.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:48.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.899+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:48.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.915+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:48.932 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:48.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:49.940 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:49.931+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:49.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:49.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:49.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:49.955+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:49.985 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:49.983+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:50.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:50.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.015+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:50.246 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:50.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:50.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:50.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:50.478 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.475+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:50.724 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.723+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:50.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:50.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:50.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.751+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:50.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:50.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:50.814 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:50.811+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:50.855 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:24:50.856 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:24:50.907 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2813,"stamp":"2023-10-14T01:24:49.882932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29431040856,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2535002,"num_read_kb":9888147,"num_write":2495055,"num_write_kb":9980621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80522,"ondisk_log_size":80522,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25015396,"kb_used_data":19957832,"kb_used_omap":1635,"kb_used_meta":5055900,"kb_avail":258100124,"statfs":{"total":289910292480,"available":264294526976,"internally_reserved":0,"allocated":20436819968,"data_stored":24746753529,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1675199,"internal_metadata":5177241665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,2,4,2,7,16,16,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16818176,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4012,"num_read_kb":16008,"num_write":4116,"num_write_kb":16464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002005"},"pg_stats":[{"pgid":"2.7","version":"155'335227","reported_seq":657722,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:46.942788+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:24:46.942788+0000","last_peered":"2023-10-14T01:24:46.942788+0000","last_clean":"2023-10-14T01:24:46.942788+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:24:46.942788+0000","last_undegraded":"2023-10-14T01:24:46.942788+0000","last_fullsized":"2023-10-14T01:24:46.942788+0000","mapping_epoch":13,"log_start":"153'325202","ondisk_log_start":"153'325202","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":595,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 312s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3723436470,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":323142,"num_read_kb":1254899,"num_write":315398,"num_write_kb":1261350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"155'335125","reported_seq":657907,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:46.952333+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:24:46.952333+0000","last_peered":"2023-10-14T01:24:46.952333+0000","last_clean":"2023-10-14T01:24:46.952333+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:24:46.952333+0000","last_undegraded":"2023-10-14T01:24:46.952333+0000","last_fullsized":"2023-10-14T01:24:46.952333+0000","mapping_epoch":13,"log_start":"153'325113","ondisk_log_start":"153'325113","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3714388336,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":324889,"num_read_kb":1251889,"num_write":314949,"num_write_kb":1259423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"155'303275","reported_seq":603372,"reported_epoch":155,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:46.952576+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:24:46.952576+0000","last_peered":"2023-10-14T01:24:46.952576+0000","last_clean":"2023-10-14T01:24:46.952576+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:24:46.952576+0000","last_undegraded":"2023-10-14T01:24:46.952576+0000","last_fullsized":"2023-10-14T01:24:46.952576+0000","mapping_epoch":13,"log_start":"152'293208","ondisk_log_start":"152'293208","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":910,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 325s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3555565970,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":300349,"num_read_kb":1185292,"num_write":302148,"num_write_kb":1208562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"155'295955","reported_seq":592080,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:46.906768+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:24:46.906768+0000","last_peered":"2023-10-14T01:24:46.906768+0000","last_clean":"2023-10-14T01:24:46.906768+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:24:46.906768+0000","last_undegraded":"2023-10-14T01:24:46.906768+0000","last_fullsized":"2023-10-14T01:24:46.906768+0000","mapping_epoch":13,"log_start":"152'285907","ondisk_log_start":"152'285907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3477733507,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":296366,"num_read_kb":1167641,"num_write":294679,"num_write_kb":1178638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"155'334108","reported_seq":668650,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:47.095267+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:24:47.095267+0000","last_peered":"2023-10-14T01:24:47.095267+0000","last_clean":"2023-10-14T01:24:47.095267+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:24:47.095267+0000","last_undegraded":"2023-10-14T01:24:47.095267+0000","last_fullsized":"2023-10-14T01:24:47.095267+0000","mapping_epoch":13,"log_start":"153'324009","ondisk_log_start":"153'324009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3913605439,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":334840,"num_read_kb":1313358,"num_write":331581,"num_write_kb":1326177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"155'321505","reported_seq":637918,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:47.288825+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:24:47.288825+0000","last_peered":"2023-10-14T01:24:47.288825+0000","last_clean":"2023-10-14T01:24:47.288825+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:24:47.288825+0000","last_undegraded":"2023-10-14T01:24:47.288825+0000","last_fullsized":"2023-10-14T01:24:47.288825+0000","mapping_epoch":13,"log_start":"152'311409","ondisk_log_start":"152'311409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3669475597,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":316918,"num_read_kb":1230864,"num_write":311650,"num_write_kb":1246386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"155'326915","reported_seq":651170,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:47.260832+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:24:47.260832+0000","last_peered":"2023-10-14T01:24:47.260832+0000","last_clean":"2023-10-14T01:24:47.260832+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:24:47.260832+0000","last_undegraded":"2023-10-14T01:24:47.260832+0000","last_fullsized":"2023-10-14T01:24:47.260832+0000","mapping_epoch":13,"log_start":"153'316911","ondisk_log_start":"153'316911","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3722088697,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":324813,"num_read_kb":1256236,"num_write":314971,"num_write_kb":1259610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"155'320353","reported_seq":633599,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:46.894328+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:24:46.894328+0000","last_peered":"2023-10-14T01:24:46.894328+0000","last_clean":"2023-10-14T01:24:46.894328+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:24:46.894328+0000","last_undegraded":"2023-10-14T01:24:46.894328+0000","last_fullsized":"2023-10-14T01:24:46.894328+0000","mapping_epoch":13,"log_start":"152'310309","ondisk_log_start":"152'310309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3654156472,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":313559,"num_read_kb":1227859,"num_write":309510,"num_write_kb":1237927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":369,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:23:49.876368+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:23:49.876368+0000","last_peered":"2023-10-14T01:23:49.876368+0000","last_clean":"2023-10-14T01:23:49.876368+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:23:49.876368+0000","last_undegraded":"2023-10-14T01:23:49.876368+0000","last_fullsized":"2023-10-14T01:23:49.876368+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29430450488,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2534876,"num_read_kb":9888038,"num_write":2494886,"num_write_kb":9978073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20435087360,"data_stored":24746566256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80395,"ondisk_log_size":80395,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739430,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5682852,"kb_used_data":3744860,"kb_used_omap":317,"kb_used_meta":1937666,"kb_avail":88688988,"statfs":{"total":96636764160,"available":90817523712,"internally_reserved":0,"allocated":3834736640,"data_stored":4643838593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":324970,"internal_metadata":1984170646},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.81100000000000005,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":12.356,"5min":12.356,"15min":12.356},"last":3.8519999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.56799999999999995,"15min":0.69499999999999995},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":0.46200000000000002,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":3.802}]},{"osd":1,"last update":"Sat Oct 14 01:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.208,"15min":1.272},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":7.8200000000000003,"5min":14.005000000000001,"15min":14.005000000000001},"last":3.8279999999999998},{"interface":"front","average":{"1min":1.7829999999999999,"5min":1.264,"15min":1.4890000000000001},"min":{"1min":0.28999999999999998,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":9.7460000000000004,"5min":9.9329999999999998,"15min":14.586},"last":0.34899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739414,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12214924,"kb_used_data":9977856,"kb_used_omap":822,"kb_used_meta":2236233,"kb_avail":82156916,"statfs":{"total":96636764160,"available":84128681984,"internally_reserved":0,"allocated":10217324544,"data_stored":12372726555,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":842494,"internal_metadata":2289902850},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,1,0,4,8,8,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.70799999999999996,"15min":0.83699999999999997},"min":{"1min":0.22,"5min":0.183,"15min":0.183},"max":{"1min":3.9620000000000002,"5min":6.1079999999999997,"15min":23.434000000000001},"last":5},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.89200000000000002,"15min":1.042},"min":{"1min":0.251,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":3.9350000000000001,"5min":8.6460000000000008,"15min":24.908000000000001},"last":5.0510000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:24:03 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4350000000000001,"15min":1.3140000000000001},"min":{"1min":0.26400000000000001,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.2530000000000001,"5min":27.837,"15min":27.837},"last":5.093},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.591,"15min":1.454},"min":{"1min":0.246,"5min":0.215,"15min":0.16800000000000001},"max":{"1min":10.411,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739428,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7117620,"kb_used_data":6235116,"kb_used_omap":495,"kb_used_meta":882000,"kb_avail":87254220,"statfs":{"total":96636764160,"available":89348321280,"internally_reserved":0,"allocated":6384758784,"data_stored":7730188381,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":903168169},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,7,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.85899999999999999,"15min":0.96299999999999997},"min":{"1min":0.185,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":3.6309999999999998,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.64100000000000001,"15min":0.61199999999999999},"min":{"1min":0.219,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.7619999999999996,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.70899999999999996,"15min":0.89200000000000002},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.1890000000000001,"5min":7.1970000000000001,"15min":9.8059999999999992},"last":0.372},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.50600000000000001,"15min":0.71199999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":1.7070000000000001,"5min":3.6789999999999998,"15min":8.9710000000000001},"last":4.141}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6383898624,"data_stored":7729665658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10217345024,"data_stored":12373084472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3833843712,"data_stored":4643816126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:24:50.908 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:50.909 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:50.909 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:51.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:51.056 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:51.070 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:51.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:51.071 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.067+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:51.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:51.098 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.095+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:51.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:51.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:52.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:52.283+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:52.289 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:52.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:52.304 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:52.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:52.305 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:52.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:52.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:52.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:24:52.306 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:52.303+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:57.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:24:57.335 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.327+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:24:57.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:57.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:57.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:57.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:57.420 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.419+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:57.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:57.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:57.440 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.439+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:57.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:57.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:57.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:57.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:57.468 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.467+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:57.499 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:57.500 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.495+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:57.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:57.519+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:58.291 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:58.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:24:58.292 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:58.287+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:58.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:58.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:58.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:58.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:58.316 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:58.315+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:58.332 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:58.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:58.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:58.323+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:58.431 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:58.431+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:58.439 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:58.435+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:59.468 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:59.467+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:24:59.475 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:59.475+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:24:59.483 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:59.483+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:24:59.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:59.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:24:59.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:24:59.495+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:24:59.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:24:59.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.523+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:59.532 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:24:59.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:24:59.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.531+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:59.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:24:59.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:24:59.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:24:59.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.611+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:24:59.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:24:59.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:24:59.635+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:01.719 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:01.715+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:01.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:01.723+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:01.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:01.747+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:01.783 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:01.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:01.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:01.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:01.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:03.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:03.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:03.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:03.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:03.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:03.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:03.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:03.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:03.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:03.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:03.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:03.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:04.331 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:04.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:04.342 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:04.338+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:04.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:04.346+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:04.353 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:04.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:04.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:04.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:04.354 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:04.350+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:06.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:06.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:06.407 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:06.398+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:06.415 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:06.410+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:09.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:09.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:09.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:09.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:09.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:09.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:09.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:09.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:09.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:09.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:10.910 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:11.333 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:11.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:11.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:11.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:11.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:11.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:11.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:11.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:11.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:11.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:11.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:11.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:11.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:11.378+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:11.816 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:25:11.816 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:25:11.866 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2824,"stamp":"2023-10-14T01:25:09.890281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29465045848,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2542858,"num_read_kb":9919479,"num_write":2503386,"num_write_kb":10013945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80600,"ondisk_log_size":80600,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24097372,"kb_used_data":20025152,"kb_used_omap":1639,"kb_used_meta":4070552,"kb_avail":259018148,"statfs":{"total":289910292480,"available":265234583552,"internally_reserved":0,"allocated":20505755648,"data_stored":24815676351,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1678558,"internal_metadata":4168246050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,10,10,12,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16846848,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3881,"num_read_kb":15480,"num_write":4130,"num_write_kb":16520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005706"},"pg_stats":[{"pgid":"2.7","version":"156'336397","reported_seq":659885,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:07.526471+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:25:07.526471+0000","last_peered":"2023-10-14T01:25:07.526471+0000","last_clean":"2023-10-14T01:25:07.526471+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:25:07.526471+0000","last_undegraded":"2023-10-14T01:25:07.526471+0000","last_fullsized":"2023-10-14T01:25:07.526471+0000","mapping_epoch":13,"log_start":"153'326302","ondisk_log_start":"153'326302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":629,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 333s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3727868342,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":324113,"num_read_kb":1258779,"num_write":316483,"num_write_kb":1265690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"156'336269","reported_seq":660050,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.533187+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:25:07.533187+0000","last_peered":"2023-10-14T01:25:07.533187+0000","last_clean":"2023-10-14T01:25:07.533187+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:25:07.533187+0000","last_undegraded":"2023-10-14T01:25:07.533187+0000","last_fullsized":"2023-10-14T01:25:07.533187+0000","mapping_epoch":13,"log_start":"153'326213","ondisk_log_start":"153'326213","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3718803824,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":325871,"num_read_kb":1255813,"num_write":316029,"num_write_kb":1263743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"156'304320","reported_seq":605421,"reported_epoch":156,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:07.528527+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:25:07.528527+0000","last_peered":"2023-10-14T01:25:07.528527+0000","last_clean":"2023-10-14T01:25:07.528527+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:25:07.528527+0000","last_undegraded":"2023-10-14T01:25:07.528527+0000","last_fullsized":"2023-10-14T01:25:07.528527+0000","mapping_epoch":13,"log_start":"153'294308","ondisk_log_start":"153'294308","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":942,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 346s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3559772562,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":301344,"num_read_kb":1189260,"num_write":303175,"num_write_kb":1212670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"156'296967","reported_seq":594013,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.528184+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:25:07.528184+0000","last_peered":"2023-10-14T01:25:07.528184+0000","last_clean":"2023-10-14T01:25:07.528184+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:25:07.528184+0000","last_undegraded":"2023-10-14T01:25:07.528184+0000","last_fullsized":"2023-10-14T01:25:07.528184+0000","mapping_epoch":13,"log_start":"153'286907","ondisk_log_start":"153'286907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3481886851,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":297292,"num_read_kb":1171329,"num_write":295691,"num_write_kb":1182686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"156'335181","reported_seq":670827,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.368746+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:25:07.368746+0000","last_peered":"2023-10-14T01:25:07.368746+0000","last_clean":"2023-10-14T01:25:07.368746+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:25:07.368746+0000","last_undegraded":"2023-10-14T01:25:07.368746+0000","last_fullsized":"2023-10-14T01:25:07.368746+0000","mapping_epoch":13,"log_start":"153'325109","ondisk_log_start":"153'325109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3918000447,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":335936,"num_read_kb":1317722,"num_write":332651,"num_write_kb":1330457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"156'322471","reported_seq":639834,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.320109+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:25:07.320109+0000","last_peered":"2023-10-14T01:25:07.320109+0000","last_clean":"2023-10-14T01:25:07.320109+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:25:07.320109+0000","last_undegraded":"2023-10-14T01:25:07.320109+0000","last_fullsized":"2023-10-14T01:25:07.320109+0000","mapping_epoch":13,"log_start":"153'312409","ondisk_log_start":"153'312409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3673334029,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":317850,"num_read_kb":1234576,"num_write":312596,"num_write_kb":1250170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"156'328032","reported_seq":653282,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.320624+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:25:07.320624+0000","last_peered":"2023-10-14T01:25:07.320624+0000","last_clean":"2023-10-14T01:25:07.320624+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:25:07.320624+0000","last_undegraded":"2023-10-14T01:25:07.320624+0000","last_fullsized":"2023-10-14T01:25:07.320624+0000","mapping_epoch":13,"log_start":"153'318011","ondisk_log_start":"153'318011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3726512377,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":325783,"num_read_kb":1260108,"num_write":316063,"num_write_kb":1263978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"156'321404","reported_seq":635641,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.526742+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:25:07.526742+0000","last_peered":"2023-10-14T01:25:07.526742+0000","last_clean":"2023-10-14T01:25:07.526742+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:25:07.526742+0000","last_undegraded":"2023-10-14T01:25:07.526742+0000","last_fullsized":"2023-10-14T01:25:07.526742+0000","mapping_epoch":13,"log_start":"153'311309","ondisk_log_start":"153'311309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3658277048,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":314543,"num_read_kb":1231783,"num_write":310529,"num_write_kb":1242003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":370,"reported_epoch":155,"state":"active+clean","last_fresh":"2023-10-14T01:24:10.664411+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:24:10.664411+0000","last_peered":"2023-10-14T01:24:10.664411+0000","last_clean":"2023-10-14T01:24:10.664411+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:24:10.664411+0000","last_undegraded":"2023-10-14T01:24:10.664411+0000","last_fullsized":"2023-10-14T01:24:10.664411+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29464455480,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2542732,"num_read_kb":9919370,"num_write":2503217,"num_write_kb":10011397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20503252992,"data_stored":24814731888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80473,"ondisk_log_size":80473,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739434,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5275788,"kb_used_data":3756996,"kb_used_omap":318,"kb_used_meta":1518465,"kb_avail":89096052,"statfs":{"total":96636764160,"available":91234357248,"internally_reserved":0,"allocated":3847163904,"data_stored":4656261571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1554908166},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,3,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.81100000000000005,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":12.356,"5min":12.356,"15min":12.356},"last":1.516},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.56799999999999995,"15min":0.69499999999999995},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":0.46200000000000002,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":1.0780000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.208,"15min":1.272},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":7.8200000000000003,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.7829999999999999,"5min":1.264,"15min":1.4890000000000001},"min":{"1min":0.28999999999999998,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":9.7460000000000004,"5min":9.9329999999999998,"15min":14.586},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739418,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11603144,"kb_used_data":10012604,"kb_used_omap":824,"kb_used_meta":1589703,"kb_avail":82768696,"statfs":{"total":96636764160,"available":84755144704,"internally_reserved":0,"allocated":10252906496,"data_stored":12408304221,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":844173,"internal_metadata":1627856499},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,5,8,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.70799999999999996,"15min":0.83699999999999997},"min":{"1min":0.22,"5min":0.183,"15min":0.183},"max":{"1min":3.9620000000000002,"5min":6.1079999999999997,"15min":23.434000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.89200000000000002,"15min":1.042},"min":{"1min":0.251,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":3.9350000000000001,"5min":8.6460000000000008,"15min":24.908000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:25:04 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.49,"15min":1.286},"min":{"1min":0.27400000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.0300000000000002,"5min":27.837,"15min":27.837},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.5309999999999999,"15min":1.403},"min":{"1min":0.22,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":4.1820000000000004,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.24299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7218440,"kb_used_data":6255552,"kb_used_omap":495,"kb_used_meta":962384,"kb_avail":87153400,"statfs":{"total":96636764160,"available":89245081600,"internally_reserved":0,"allocated":6405685248,"data_stored":7751110559,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":985481385},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.83299999999999996,"15min":0.94599999999999995},"min":{"1min":0.21299999999999999,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":2.2170000000000001,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.61299999999999999},"min":{"1min":0.20999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.8380000000000001,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.66100000000000003,"15min":0.86499999999999999},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.6720000000000002,"5min":5.1890000000000001,"15min":9.8059999999999992},"last":5.4610000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.56699999999999995,"15min":0.71599999999999997},"min":{"1min":0.23899999999999999,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":4.141,"5min":4.141,"15min":8.9710000000000001},"last":0.67400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6405230592,"data_stored":7750997626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10251784192,"data_stored":12407523640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3846238208,"data_stored":4656210622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:25:11.870 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:11.870 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:11.871 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:12.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:25:12.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.842+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:25:12.873 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.870+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:12.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.886+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:12.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:12.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:12.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:12.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:12.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:12.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:12.898+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:13.908 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:13.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:13.923 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:13.918+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:13.928 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:13.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:13.966 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:13.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:13.984 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:13.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:14.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:14.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:14.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:14.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:14.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:14.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:15.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:15.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:15.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:15.086+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:15.103 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:15.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:15.132 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:15.130+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:16.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:16.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:16.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:16.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:16.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:16.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:16.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:16.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:16.764 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:16.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:16.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:16.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:17.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:17.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:17.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:17.390 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.382+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:17.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:17.416 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:17.810 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:17.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:17.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:17.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:17.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:19.095 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:19.090+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:19.115 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:19.114+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:19.129 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:19.126+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:19.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:19.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:19.137 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:19.134+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:22.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:22.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:22.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:22.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:22.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:22.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:23.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:23.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:23.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:23.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:23.274 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:23.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:23.297 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:23.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:23.304 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:23.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:23.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:23.310+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:23.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:23.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:25:23.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:23.858+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:23.867 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:23.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:23.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:23.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:23.883 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:23.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:23.884 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:23.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:23.884 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:23.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:25.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:25.846+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:25.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:25.862+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:25.869 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:25.866+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:25.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:25.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:25.882 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:25.878+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:25.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:25.942+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:25.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:25.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:25.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:25.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:25.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:25.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:25.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:25.970+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:26.680 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.678+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:26.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:26.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:26.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:26.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.710+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:26.737 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:26.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:26.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:26.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:30.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:30.966+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:31.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:31.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:31.872 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:32.387 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:25:32.388 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:25:32.409 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2836,"stamp":"2023-10-14T01:25:31.901069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29499681624,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2550772,"num_read_kb":9951055,"num_write":2511881,"num_write_kb":10047925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80439,"ondisk_log_size":80439,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24430884,"kb_used_data":20092168,"kb_used_omap":1639,"kb_used_meta":4337048,"kb_avail":258684636,"statfs":{"total":289910292480,"available":264893067264,"internally_reserved":0,"allocated":20574380032,"data_stored":24884287748,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1678558,"internal_metadata":4441137954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,7,12,13,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17481728,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3980,"num_read_kb":15884,"num_write":4265,"num_write_kb":17060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006528"},"pg_stats":[{"pgid":"2.7","version":"157'337522","reported_seq":662008,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:28.084770+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:25:28.084770+0000","last_peered":"2023-10-14T01:25:28.084770+0000","last_clean":"2023-10-14T01:25:28.084770+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:25:28.084770+0000","last_undegraded":"2023-10-14T01:25:28.084770+0000","last_fullsized":"2023-10-14T01:25:28.084770+0000","mapping_epoch":13,"log_start":"153'327502","ondisk_log_start":"153'327502","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":690,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 353s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3732074934,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":325089,"num_read_kb":1262679,"num_write":317519,"num_write_kb":1269834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'337459","reported_seq":662271,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:28.092492+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:25:28.092492+0000","last_peered":"2023-10-14T01:25:28.092492+0000","last_clean":"2023-10-14T01:25:28.092492+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:25:28.092492+0000","last_undegraded":"2023-10-14T01:25:28.092492+0000","last_fullsized":"2023-10-14T01:25:28.092492+0000","mapping_epoch":13,"log_start":"153'327413","ondisk_log_start":"153'327413","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3723432304,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":326891,"num_read_kb":1259869,"num_write":317163,"num_write_kb":1268279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'305350","reported_seq":607455,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:28.045422+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:25:28.045422+0000","last_peered":"2023-10-14T01:25:28.045422+0000","last_clean":"2023-10-14T01:25:28.045422+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:25:28.045422+0000","last_undegraded":"2023-10-14T01:25:28.045422+0000","last_fullsized":"2023-10-14T01:25:28.045422+0000","mapping_epoch":13,"log_start":"153'295308","ondisk_log_start":"153'295308","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":990,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 366s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3563835794,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":302335,"num_read_kb":1193216,"num_write":304178,"num_write_kb":1216682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'298012","reported_seq":595965,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:28.049977+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:25:28.049977+0000","last_peered":"2023-10-14T01:25:28.049977+0000","last_clean":"2023-10-14T01:25:28.049977+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:25:28.049977+0000","last_undegraded":"2023-10-14T01:25:28.049977+0000","last_fullsized":"2023-10-14T01:25:28.049977+0000","mapping_epoch":13,"log_start":"153'288007","ondisk_log_start":"153'288007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3486171267,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":298192,"num_read_kb":1174913,"num_write":296736,"num_write_kb":1186866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"157'336320","reported_seq":673022,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:27.590538+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:25:27.590538+0000","last_peered":"2023-10-14T01:25:27.590538+0000","last_clean":"2023-10-14T01:25:27.590538+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:25:27.590538+0000","last_undegraded":"2023-10-14T01:25:27.590538+0000","last_fullsized":"2023-10-14T01:25:27.590538+0000","mapping_epoch":13,"log_start":"153'326309","ondisk_log_start":"153'326309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3922645311,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":336989,"num_read_kb":1321922,"num_write":333782,"num_write_kb":1334981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"157'323529","reported_seq":641832,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:27.421698+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:25:27.421698+0000","last_peered":"2023-10-14T01:25:27.421698+0000","last_clean":"2023-10-14T01:25:27.421698+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:25:27.421698+0000","last_undegraded":"2023-10-14T01:25:27.421698+0000","last_fullsized":"2023-10-14T01:25:27.421698+0000","mapping_epoch":13,"log_start":"153'313509","ondisk_log_start":"153'313509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3677557005,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":318777,"num_read_kb":1238280,"num_write":313638,"num_write_kb":1254338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"157'329085","reported_seq":655349,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:27.395204+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:25:27.395204+0000","last_peered":"2023-10-14T01:25:27.395204+0000","last_clean":"2023-10-14T01:25:27.395204+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:25:27.395204+0000","last_undegraded":"2023-10-14T01:25:27.395204+0000","last_fullsized":"2023-10-14T01:25:27.395204+0000","mapping_epoch":13,"log_start":"153'319011","ondisk_log_start":"153'319011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3730718969,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":326781,"num_read_kb":1264096,"num_write":317096,"num_write_kb":1268110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"157'322503","reported_seq":637800,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:28.062802+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:25:28.062802+0000","last_peered":"2023-10-14T01:25:28.062802+0000","last_clean":"2023-10-14T01:25:28.062802+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:25:28.062802+0000","last_undegraded":"2023-10-14T01:25:28.062802+0000","last_fullsized":"2023-10-14T01:25:28.062802+0000","mapping_epoch":13,"log_start":"153'312409","ondisk_log_start":"153'312409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3662655672,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":315592,"num_read_kb":1235971,"num_write":311600,"num_write_kb":1246287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":371,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:04.894502+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:25:04.894502+0000","last_peered":"2023-10-14T01:25:04.894502+0000","last_clean":"2023-10-14T01:25:04.894502+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:25:04.894502+0000","last_undegraded":"2023-10-14T01:25:04.894502+0000","last_fullsized":"2023-10-14T01:25:04.894502+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29499091256,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2550646,"num_read_kb":9950946,"num_write":2511712,"num_write_kb":10045377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20572758016,"data_stored":24884236912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80312,"ondisk_log_size":80312,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739438,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5342368,"kb_used_data":3769432,"kb_used_omap":318,"kb_used_meta":1572609,"kb_avail":89029472,"statfs":{"total":96636764160,"available":91166179328,"internally_reserved":0,"allocated":3859898368,"data_stored":4668991706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1610351622},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:37 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":0.81100000000000005,"15min":0.77600000000000002},"min":{"1min":0.191,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":12.356,"5min":12.356,"15min":12.356},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.56799999999999995,"15min":0.69499999999999995},"min":{"1min":0.23200000000000001,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":0.46200000000000002,"5min":7.3339999999999996,"15min":7.6680000000000001},"last":0.21199999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:24:37 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.208,"15min":1.272},"min":{"1min":0.23000000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":7.8200000000000003,"5min":14.005000000000001,"15min":14.005000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.7829999999999999,"5min":1.264,"15min":1.4890000000000001},"min":{"1min":0.28999999999999998,"5min":0.23899999999999999,"15min":0.20899999999999999},"max":{"1min":9.7460000000000004,"5min":9.9329999999999998,"15min":14.586},"last":0.438}]}]},{"osd":1,"up_from":8,"seq":34359739422,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11771124,"kb_used_data":10045996,"kb_used_omap":824,"kb_used_meta":1724295,"kb_avail":82600716,"statfs":{"total":96636764160,"available":84583133184,"internally_reserved":0,"allocated":10287099904,"data_stored":12442493300,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":844173,"internal_metadata":1765678707},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,8,11,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.70799999999999996,"15min":0.83699999999999997},"min":{"1min":0.22,"5min":0.183,"15min":0.183},"max":{"1min":3.9620000000000002,"5min":6.1079999999999997,"15min":23.434000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.89200000000000002,"15min":1.042},"min":{"1min":0.251,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":3.9350000000000001,"5min":8.6460000000000008,"15min":24.908000000000001},"last":0.307}]},{"osd":2,"last update":"Sat Oct 14 01:25:04 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.49,"15min":1.286},"min":{"1min":0.27400000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.0300000000000002,"5min":27.837,"15min":27.837},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.5309999999999999,"15min":1.403},"min":{"1min":0.22,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":4.1820000000000004,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739436,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7317392,"kb_used_data":6276740,"kb_used_omap":495,"kb_used_meta":1040144,"kb_avail":87054448,"statfs":{"total":96636764160,"available":89143754752,"internally_reserved":0,"allocated":6427381760,"data_stored":7772802742,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1065107625},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.83299999999999996,"15min":0.94599999999999995},"min":{"1min":0.21299999999999999,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":2.2170000000000001,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.432},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71099999999999997,"15min":0.61299999999999999},"min":{"1min":0.20999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.8380000000000001,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.66100000000000003,"15min":0.86499999999999999},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.6720000000000002,"5min":5.1890000000000001,"15min":9.8059999999999992},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.56699999999999995,"15min":0.71599999999999997},"min":{"1min":0.23899999999999999,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":4.141,"5min":4.141,"15min":8.9710000000000001},"last":0.29499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6426832896,"data_stored":7772599930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10286911488,"data_stored":12442650936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3859013632,"data_stored":4668986046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:25:32.411 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:32.411 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:32.412 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:33.782 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:33.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:33.783 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:33.778+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:33.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:33.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:33.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:33.798+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:33.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:33.834+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:33.984 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:33.978+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:34.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:34.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:34.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:34.041 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.038+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:34.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:34.522+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:34.887 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.882+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:34.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.910+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:34.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.910+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:34.933 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:34.910+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:36.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:36.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:36.673 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:36.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:37.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:37.104 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:37.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:37.105 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.098+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:37.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.106+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:37.303 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:37.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:37.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:37.311 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:37.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:37.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:37.827 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:37.826+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:37.836 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:37.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:37.875 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:37.870+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:37.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:37.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:37.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:37.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:37.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:37.894+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:41.131 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:41.130+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:41.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:41.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:41.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:41.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:41.162 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:41.158+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:41.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:41.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:41.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:41.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:41.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:41.862+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:41.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:41.890+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:41.904 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:41.902+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:42.925 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:42.922+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:42.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:42.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:42.973 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:42.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:42.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:42.962+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:43.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.010+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:43.019 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:43.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:43.545 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:43.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:25:43.546 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:43.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:43.552 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:43.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:43.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:43.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:43.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:43.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:43.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:43.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:43.553 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:43.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:43.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:43.766 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:43.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:43.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:43.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:43.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:44.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:44.142+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:44.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:44.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:44.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:44.190+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:44.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:44.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:44.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:44.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:44.986 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:44.982+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:47.739 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:47.734+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:47.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:47.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:47.792 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:47.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:47.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:47.786+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:50.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.198+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:50.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:50.224 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:50.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:50.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.214+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:50.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:50.225 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:50.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.222+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:50.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:50.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:50.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:50.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:50.342 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.338+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:50.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:50.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.414+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:50.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:50.426+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:50.496 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:50.494+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:50.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:50.494+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:50.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:50.518+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:50.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:50.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:50.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:50.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:50.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:50.538+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:52.413 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:53.409 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:25:53.410 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:25:53.456 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2846,"stamp":"2023-10-14T01:25:51.906735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29533711192,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2558578,"num_read_kb":9982227,"num_write":2520221,"num_write_kb":10081285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80418,"ondisk_log_size":80418,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24858404,"kb_used_data":20158724,"kb_used_omap":1639,"kb_used_meta":4698008,"kb_avail":258257116,"statfs":{"total":289910292480,"available":264455286784,"internally_reserved":0,"allocated":20642533376,"data_stored":24952441092,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1678558,"internal_metadata":4810760994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,6,11,10,16,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17342464,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3859,"num_read_kb":15420,"num_write":4249,"num_write_kb":16996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004283"},"pg_stats":[{"pgid":"2.7","version":"157'338659","reported_seq":664140,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:48.758165+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:25:48.758165+0000","last_peered":"2023-10-14T01:25:48.758165+0000","last_clean":"2023-10-14T01:25:48.758165+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:25:48.758165+0000","last_undegraded":"2023-10-14T01:25:48.758165+0000","last_fullsized":"2023-10-14T01:25:48.758165+0000","mapping_epoch":13,"log_start":"154'328602","ondisk_log_start":"154'328602","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":734,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 374s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3736408502,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":326057,"num_read_kb":1266543,"num_write":318580,"num_write_kb":1274078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'338625","reported_seq":664460,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:48.741818+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:25:48.741818+0000","last_peered":"2023-10-14T01:25:48.741818+0000","last_clean":"2023-10-14T01:25:48.741818+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:25:48.741818+0000","last_undegraded":"2023-10-14T01:25:48.741818+0000","last_fullsized":"2023-10-14T01:25:48.741818+0000","mapping_epoch":13,"log_start":"154'328613","ondisk_log_start":"154'328613","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3728015728,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":327893,"num_read_kb":1263869,"num_write":318279,"num_write_kb":1272743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'306419","reported_seq":609428,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:48.756517+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:25:48.756517+0000","last_peered":"2023-10-14T01:25:48.756517+0000","last_clean":"2023-10-14T01:25:48.756517+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:25:48.756517+0000","last_undegraded":"2023-10-14T01:25:48.756517+0000","last_fullsized":"2023-10-14T01:25:48.756517+0000","mapping_epoch":13,"log_start":"153'296408","ondisk_log_start":"153'296408","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1046,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 387s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3568038290,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":303233,"num_read_kb":1196804,"num_write":305213,"num_write_kb":1220822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'298971","reported_seq":597880,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:48.742520+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:25:48.742520+0000","last_peered":"2023-10-14T01:25:48.742520+0000","last_clean":"2023-10-14T01:25:48.742520+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:25:48.742520+0000","last_undegraded":"2023-10-14T01:25:48.742520+0000","last_fullsized":"2023-10-14T01:25:48.742520+0000","mapping_epoch":13,"log_start":"153'288907","ondisk_log_start":"153'288907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3490074755,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":299150,"num_read_kb":1178745,"num_write":297695,"num_write_kb":1190702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"157'337436","reported_seq":675165,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.729255+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:25:47.729255+0000","last_peered":"2023-10-14T01:25:47.729255+0000","last_clean":"2023-10-14T01:25:47.729255+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:25:47.729255+0000","last_undegraded":"2023-10-14T01:25:47.729255+0000","last_fullsized":"2023-10-14T01:25:47.729255+0000","mapping_epoch":13,"log_start":"154'327409","ondisk_log_start":"154'327409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3927224639,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":338013,"num_read_kb":1326006,"num_write":334897,"num_write_kb":1339441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"157'324527","reported_seq":643779,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.524566+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:25:47.524566+0000","last_peered":"2023-10-14T01:25:47.524566+0000","last_clean":"2023-10-14T01:25:47.524566+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:25:47.524566+0000","last_undegraded":"2023-10-14T01:25:47.524566+0000","last_fullsized":"2023-10-14T01:25:47.524566+0000","mapping_epoch":13,"log_start":"153'314509","ondisk_log_start":"153'314509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3681497357,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":319710,"num_read_kb":1242008,"num_write":314612,"num_write_kb":1258234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"157'330159","reported_seq":657433,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.503657+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:25:47.503657+0000","last_peered":"2023-10-14T01:25:47.503657+0000","last_clean":"2023-10-14T01:25:47.503657+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:25:47.503657+0000","last_undegraded":"2023-10-14T01:25:47.503657+0000","last_fullsized":"2023-10-14T01:25:47.503657+0000","mapping_epoch":13,"log_start":"153'320111","ondisk_log_start":"153'320111","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3734966521,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":327773,"num_read_kb":1268064,"num_write":318140,"num_write_kb":1272286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"157'323563","reported_seq":639899,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:25:48.731607+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:25:48.731607+0000","last_peered":"2023-10-14T01:25:48.731607+0000","last_clean":"2023-10-14T01:25:48.731607+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:25:48.731607+0000","last_undegraded":"2023-10-14T01:25:48.731607+0000","last_fullsized":"2023-10-14T01:25:48.731607+0000","mapping_epoch":13,"log_start":"153'313509","ondisk_log_start":"153'313509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3666895032,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":316623,"num_read_kb":1240079,"num_write":312636,"num_write_kb":1250431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":371,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:04.894502+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:25:04.894502+0000","last_peered":"2023-10-14T01:25:04.894502+0000","last_clean":"2023-10-14T01:25:04.894502+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:25:04.894502+0000","last_undegraded":"2023-10-14T01:25:04.894502+0000","last_fullsized":"2023-10-14T01:25:04.894502+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29533120824,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2558452,"num_read_kb":9982118,"num_write":2520052,"num_write_kb":10078737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20640751616,"data_stored":24952230512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80291,"ondisk_log_size":80291,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739442,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5403564,"kb_used_data":3782048,"kb_used_omap":318,"kb_used_meta":1621185,"kb_avail":88968276,"statfs":{"total":96636764160,"available":91103514624,"internally_reserved":0,"allocated":3872817152,"data_stored":4681910490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1660093446},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:39 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":0.79500000000000004,"15min":0.82399999999999995},"min":{"1min":0.191,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":5.7560000000000002,"5min":12.356,"15min":12.356},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.53300000000000003,"15min":0.69399999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":4.6959999999999997,"5min":5.6230000000000002,"15min":7.6680000000000001},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:25:39 2023","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.1279999999999999,"15min":1.3320000000000001},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":5.8949999999999996,"5min":8.6159999999999997,"15min":14.005000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.577,"5min":1.347,"15min":1.5189999999999999},"min":{"1min":0.28999999999999998,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":7.0570000000000004,"5min":9.9329999999999998,"15min":14.586},"last":0.32100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739426,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11996016,"kb_used_data":10079204,"kb_used_omap":824,"kb_used_meta":1915975,"kb_avail":82375824,"statfs":{"total":96636764160,"available":84352843776,"internally_reserved":0,"allocated":10321104896,"data_stored":12476498292,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":844173,"internal_metadata":1961959027},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1,8,6,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.85199999999999998,"15min":0.83199999999999996},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":7.3460000000000001,"5min":7.3460000000000001,"15min":23.434000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.97999999999999998,"15min":1.016},"min":{"1min":0.245,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":5.8440000000000003,"5min":8.6460000000000008,"15min":24.908000000000001},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:25:04 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.49,"15min":1.286},"min":{"1min":0.27400000000000002,"5min":0.245,"15min":0.23499999999999999},"max":{"1min":6.0300000000000002,"5min":27.837,"15min":27.837},"last":0.314},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.5309999999999999,"15min":1.403},"min":{"1min":0.22,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":4.1820000000000004,"5min":25.710000000000001,"15min":25.710000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739440,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7458824,"kb_used_data":6297472,"kb_used_omap":495,"kb_used_meta":1160848,"kb_avail":86913016,"statfs":{"total":96636764160,"available":88998928384,"internally_reserved":0,"allocated":6448611328,"data_stored":7794032310,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1188708521},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:25:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":0.99199999999999999,"15min":1.0109999999999999},"min":{"1min":0.222,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":5.3250000000000002,"5min":6.6639999999999997,"15min":11.734999999999999},"last":2.9220000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.76000000000000001,"15min":0.63300000000000001},"min":{"1min":0.23999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":4.7000000000000002,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:25:47 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.78600000000000003,"15min":0.873},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.4610000000000003,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":2.8679999999999999},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.65400000000000003,"15min":0.64500000000000002},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":5.2699999999999996,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":2.8969999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6448066560,"data_stored":7793833594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10321293312,"data_stored":12477032760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3871391744,"data_stored":4681364158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:25:53.458 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:53.458 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:53.459 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:54.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:54.902+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:54.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:54.950+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:54.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:54.950+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:54.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:54.954+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:54.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:54.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:54.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:54.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:54.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:54.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:54.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:54.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:54.961 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:54.958+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:54.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:54.962 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:54.963 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:54.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:56.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:56.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:56.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:56.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:56.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:56.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:56.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:56.574+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:57.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:57.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:57.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:57.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:57.177 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:57.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:57.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:57.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:57.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:57.210+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:57.222 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:57.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:58.345 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:25:58.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:58.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:58.425 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.422+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:58.432 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.430+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:58.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:58.474 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.466+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:58.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:25:58.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:25:58.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:25:58.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.594+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:25:58.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.602+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:25:58.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:25:58.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:25:58.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:25:59.199 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.197+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:25:59.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.205+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:25:59.230 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.221+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:59.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:59.231 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:59.232 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:25:59.348 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.345+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:25:59.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:25:59.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:25:59.384 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:25:59.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:00.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:00.209+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:26:00.234 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:00.229+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:00.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:00.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:00.263 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:00.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:00.264 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:00.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:03.387 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:03.381+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:03.415 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:03.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:03.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:03.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:03.416 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:03.405+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:03.756 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:03.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:03.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:03.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:03.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:03.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:03.777 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:03.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:03.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:03.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:03.845 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:03.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:06.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:06.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:06.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:06.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:06.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:06.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:06.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:06.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:07.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:07.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:07.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:07.113 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:07.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:07.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:07.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:07.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:07.504 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:07.513 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:07.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:07.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:07.833 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:07.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:07.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:07.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:07.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:07.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:08.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:08.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:08.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:08.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:08.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:08.165+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:08.188 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:08.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:08.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:08.197+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:08.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:08.225+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:09.239 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:09.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:09.240 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:09.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:09.245 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:09.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:09.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:09.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:09.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:09.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:09.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:09.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:09.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:09.545+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:09.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:09.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:09.583 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:09.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:09.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:09.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:13.462 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:13.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.513+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:13.523 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:13.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:13.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:13.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:13.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.533+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:13.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:13.586 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:13.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:14.308 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:26:14.309 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:26:14.384 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2857,"stamp":"2023-10-14T01:26:13.910509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29571435352,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2567433,"num_read_kb":10017547,"num_write":2529478,"num_write_kb":10118313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80557,"ondisk_log_size":80557,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25142564,"kb_used_data":20235020,"kb_used_omap":1639,"kb_used_meta":4905880,"kb_avail":257972956,"statfs":{"total":289910292480,"available":264164306944,"internally_reserved":0,"allocated":20720660480,"data_stored":25030568196,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1678558,"internal_metadata":5023621922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,6,5,13,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20475904,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4840,"num_read_kb":19300,"num_write":5028,"num_write_kb":20112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002193"},"pg_stats":[{"pgid":"2.7","version":"157'339797","reported_seq":666313,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:09.429166+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:26:09.429166+0000","last_peered":"2023-10-14T01:26:09.429166+0000","last_clean":"2023-10-14T01:26:09.429166+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:26:09.429166+0000","last_undegraded":"2023-10-14T01:26:09.429166+0000","last_fullsized":"2023-10-14T01:26:09.429166+0000","mapping_epoch":13,"log_start":"154'329702","ondisk_log_start":"154'329702","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":780,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 395s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3740692918,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":327070,"num_read_kb":1270591,"num_write":319630,"num_write_kb":1278278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'339748","reported_seq":666658,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:09.426831+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:26:09.426831+0000","last_peered":"2023-10-14T01:26:09.426831+0000","last_clean":"2023-10-14T01:26:09.426831+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:26:09.426831+0000","last_undegraded":"2023-10-14T01:26:09.426831+0000","last_fullsized":"2023-10-14T01:26:09.426831+0000","mapping_epoch":13,"log_start":"154'329713","ondisk_log_start":"154'329713","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3732390256,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":328951,"num_read_kb":1268089,"num_write":319342,"num_write_kb":1276995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"157'307487","reported_seq":611484,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:09.307094+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:26:09.307094+0000","last_peered":"2023-10-14T01:26:09.307094+0000","last_clean":"2023-10-14T01:26:09.307094+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:26:09.307094+0000","last_undegraded":"2023-10-14T01:26:09.307094+0000","last_fullsized":"2023-10-14T01:26:09.307094+0000","mapping_epoch":13,"log_start":"154'297408","ondisk_log_start":"154'297408","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1080,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 408s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3572326802,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":304215,"num_read_kb":1200708,"num_write":306259,"num_write_kb":1225006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"157'299956","reported_seq":599780,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:09.347950+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:26:09.347950+0000","last_peered":"2023-10-14T01:26:09.347950+0000","last_clean":"2023-10-14T01:26:09.347950+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:26:09.347950+0000","last_undegraded":"2023-10-14T01:26:09.347950+0000","last_fullsized":"2023-10-14T01:26:09.347950+0000","mapping_epoch":13,"log_start":"154'289907","ondisk_log_start":"154'289907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3494072451,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":300062,"num_read_kb":1182381,"num_write":298680,"num_write_kb":1194642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"157'338830","reported_seq":677886,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:12.980197+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:26:12.980197+0000","last_peered":"2023-10-14T01:26:12.980197+0000","last_clean":"2023-10-14T01:26:12.980197+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:26:12.980197+0000","last_undegraded":"2023-10-14T01:26:12.980197+0000","last_fullsized":"2023-10-14T01:26:12.980197+0000","mapping_epoch":13,"log_start":"154'328809","ondisk_log_start":"154'328809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3932893503,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":339333,"num_read_kb":1331270,"num_write":336289,"num_write_kb":1345009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"157'325890","reported_seq":646387,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:12.720858+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:26:12.720858+0000","last_peered":"2023-10-14T01:26:12.720858+0000","last_clean":"2023-10-14T01:26:12.720858+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:26:12.720858+0000","last_undegraded":"2023-10-14T01:26:12.720858+0000","last_fullsized":"2023-10-14T01:26:12.720858+0000","mapping_epoch":13,"log_start":"154'315809","ondisk_log_start":"154'315809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3686912269,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":320937,"num_read_kb":1246900,"num_write":315940,"num_write_kb":1263546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"157'331546","reported_seq":660157,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:12.753668+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:26:12.753668+0000","last_peered":"2023-10-14T01:26:12.753668+0000","last_clean":"2023-10-14T01:26:12.753668+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:26:12.753668+0000","last_undegraded":"2023-10-14T01:26:12.753668+0000","last_fullsized":"2023-10-14T01:26:12.753668+0000","mapping_epoch":13,"log_start":"154'321511","ondisk_log_start":"154'321511","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3740385529,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":329084,"num_read_kb":1273296,"num_write":319482,"num_write_kb":1277654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"157'324644","reported_seq":642014,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:09.347093+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:26:09.347093+0000","last_peered":"2023-10-14T01:26:09.347093+0000","last_clean":"2023-10-14T01:26:09.347093+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:26:09.347093+0000","last_undegraded":"2023-10-14T01:26:09.347093+0000","last_fullsized":"2023-10-14T01:26:09.347093+0000","mapping_epoch":13,"log_start":"154'314609","ondisk_log_start":"154'314609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3671171256,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":317655,"num_read_kb":1244203,"num_write":313687,"num_write_kb":1254635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":371,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-14T01:25:04.894502+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:25:04.894502+0000","last_peered":"2023-10-14T01:25:04.894502+0000","last_clean":"2023-10-14T01:25:04.894502+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:25:04.894502+0000","last_undegraded":"2023-10-14T01:25:04.894502+0000","last_fullsized":"2023-10-14T01:25:04.894502+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29570844984,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2567307,"num_read_kb":10017438,"num_write":2529309,"num_write_kb":10115765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20718280704,"data_stored":25029759600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80430,"ondisk_log_size":80430,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5432828,"kb_used_data":3797748,"kb_used_omap":318,"kb_used_meta":1634753,"kb_avail":88939012,"statfs":{"total":96636764160,"available":91073548288,"internally_reserved":0,"allocated":3888893952,"data_stored":4697987290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1673987078},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:39 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":0.79500000000000004,"15min":0.82399999999999995},"min":{"1min":0.191,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":5.7560000000000002,"5min":12.356,"15min":12.356},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.53300000000000003,"15min":0.69399999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":4.6959999999999997,"5min":5.6230000000000002,"15min":7.6680000000000001},"last":0.24299999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:25:39 2023","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.1279999999999999,"15min":1.3320000000000001},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":5.8949999999999996,"5min":8.6159999999999997,"15min":14.005000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.577,"5min":1.347,"15min":1.5189999999999999},"min":{"1min":0.28999999999999998,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":7.0570000000000004,"5min":9.9329999999999998,"15min":14.586},"last":0.495}]}]},{"osd":1,"up_from":8,"seq":34359739430,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12125628,"kb_used_data":10113076,"kb_used_omap":824,"kb_used_meta":2011719,"kb_avail":82246212,"statfs":{"total":96636764160,"available":84220121088,"internally_reserved":0,"allocated":10355789824,"data_stored":12511183220,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":844173,"internal_metadata":2060000883},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,2,10,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.85199999999999998,"15min":0.83199999999999996},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":7.3460000000000001,"5min":7.3460000000000001,"15min":23.434000000000001},"last":2.3570000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.97999999999999998,"15min":1.016},"min":{"1min":0.245,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":5.8440000000000003,"5min":8.6460000000000008,"15min":24.908000000000001},"last":2.3919999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:26:06 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1240000000000001,"15min":1.278},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":5.7729999999999997,"5min":6.3310000000000004,"15min":27.837},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.73,"5min":1.3340000000000001,"15min":1.452},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":15.09,"5min":15.09,"15min":25.710000000000001},"last":5.7530000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739445,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7584108,"kb_used_data":6324196,"kb_used_omap":495,"kb_used_meta":1259408,"kb_avail":86787732,"statfs":{"total":96636764160,"available":88870637568,"internally_reserved":0,"allocated":6475976704,"data_stored":7821397686,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1289633961},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:25:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":0.99199999999999999,"15min":1.0109999999999999},"min":{"1min":0.222,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":5.3250000000000002,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.76000000000000001,"15min":0.63300000000000001},"min":{"1min":0.23999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":4.7000000000000002,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:25:47 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.78600000000000003,"15min":0.873},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.4610000000000003,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.65400000000000003,"15min":0.64500000000000002},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":5.2699999999999996,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6474530816,"data_stored":7820297850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10356109312,"data_stored":12511848760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3887640576,"data_stored":4697612990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:26:14.386 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:14.386 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:14.387 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:26:15.275 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:26:15.310 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-14T01:26:15.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:15.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:15.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:15.389 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:15.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:15.794 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:15.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.789+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:15.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:15.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:15.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:15.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:15.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:15.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:15.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:15.865 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:15.866 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:15.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:16.175 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-14T01:26:16.254 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:18.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:18.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:18.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:18.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:18.837 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:18.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:18.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:18.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:18.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:18.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:18.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:18.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:26:18.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:18.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:18.856 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:18.853+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:18.862 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:18.857+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:18.866 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:18.861+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:22.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:22.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:22.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:22.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:22.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:22.677+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:22.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:22.685+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:22.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:22.745+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:23.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:23.601+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:23.604 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:23.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:23.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:23.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:23.605 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:23.605+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:24.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:24.141+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:24.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:24.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:24.150 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:24.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:24.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:24.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:24.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:24.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:24.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:24.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:24.751 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.753+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:26:24.752 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.753+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:24.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.765+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:24.787 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.785+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:24.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:24.794 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:24.795 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.793+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:24.912 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.913+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:24.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.921+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:24.934 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.933+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:24.955 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:24.956 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:24.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:26.790 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:26.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:26.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:26.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:26.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:26.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:26.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:26.805+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:26.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:26.841+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:26.861 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:26.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:27.219 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:27.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:27.235 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:27.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:27.236 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:27.237+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:27.294 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:27.293+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:27.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:27.297+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:27.295 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:27.297+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:30.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:26:30.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:26:30.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:30.027 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:30.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:30.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:30.028 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:30.033 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:30.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.041+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:30.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:30.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:32.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:32.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:32.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.457+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:32.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:32.796 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.797+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:32.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:32.834 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:32.835 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:32.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:32.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:32.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:34.352 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:34.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.353+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:34.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:34.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:34.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:34.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.377+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:34.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.381+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:34.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.389+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:34.454 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:34.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:34.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.453+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:34.493 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.493+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:34.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:34.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:34.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:34.856 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:34.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:34.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:26:34.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:34.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:34.881 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:34.877+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:36.255 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:36.293 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:36.289+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:36.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:36.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:36.329 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:36.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:36.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:36.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:36.388 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:36.385+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:36.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:36.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:36.766 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:26:36.767 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:26:36.806 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2869,"stamp":"2023-10-14T01:26:35.919269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29611572056,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2577026,"num_read_kb":10055787,"num_write":2539261,"num_write_kb":10157445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80422,"ondisk_log_size":80422,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25561880,"kb_used_data":20311356,"kb_used_omap":1639,"kb_used_meta":5248856,"kb_avail":257553640,"statfs":{"total":289910292480,"available":263734927360,"internally_reserved":0,"allocated":20798828544,"data_stored":25108723144,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1678558,"internal_metadata":5374829346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,15,13,17,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17420288,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4184,"num_read_kb":16684,"num_write":4235,"num_write_kb":16940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006768"},"pg_stats":[{"pgid":"2.7","version":"158'341145","reported_seq":669014,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:35.125013+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:26:35.125013+0000","last_peered":"2023-10-14T01:26:35.125013+0000","last_clean":"2023-10-14T01:26:35.125013+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:26:35.125013+0000","last_undegraded":"2023-10-14T01:26:35.125013+0000","last_fullsized":"2023-10-14T01:26:35.125013+0000","mapping_epoch":13,"log_start":"154'331102","ondisk_log_start":"154'331102","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":839,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 420s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3745808822,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":328395,"num_read_kb":1275863,"num_write":320880,"num_write_kb":1283278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'341138","reported_seq":669438,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:35.137871+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:26:35.137871+0000","last_peered":"2023-10-14T01:26:35.137871+0000","last_clean":"2023-10-14T01:26:35.137871+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:26:35.137871+0000","last_undegraded":"2023-10-14T01:26:35.137871+0000","last_fullsized":"2023-10-14T01:26:35.137871+0000","mapping_epoch":13,"log_start":"154'331113","ondisk_log_start":"154'331113","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3737842032,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":330320,"num_read_kb":1273553,"num_write":320667,"num_write_kb":1282295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'308817","reported_seq":614008,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:35.155733+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:26:35.155733+0000","last_peered":"2023-10-14T01:26:35.155733+0000","last_clean":"2023-10-14T01:26:35.155733+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:26:35.155733+0000","last_undegraded":"2023-10-14T01:26:35.155733+0000","last_fullsized":"2023-10-14T01:26:35.155733+0000","mapping_epoch":13,"log_start":"154'298808","ondisk_log_start":"154'298808","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1152,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 434s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3577627026,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":305392,"num_read_kb":1205384,"num_write":307542,"num_write_kb":1230138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'301184","reported_seq":602218,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:35.079681+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:26:35.079681+0000","last_peered":"2023-10-14T01:26:35.079681+0000","last_clean":"2023-10-14T01:26:35.079681+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:26:35.079681+0000","last_undegraded":"2023-10-14T01:26:35.079681+0000","last_fullsized":"2023-10-14T01:26:35.079681+0000","mapping_epoch":13,"log_start":"154'291107","ondisk_log_start":"154'291107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3499102339,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":301272,"num_read_kb":1187213,"num_write":299908,"num_write_kb":1199554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"158'340001","reported_seq":680131,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.094484+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:26:33.094484+0000","last_peered":"2023-10-14T01:26:33.094484+0000","last_clean":"2023-10-14T01:26:33.094484+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:26:33.094484+0000","last_undegraded":"2023-10-14T01:26:33.094484+0000","last_fullsized":"2023-10-14T01:26:33.094484+0000","mapping_epoch":13,"log_start":"154'329909","ondisk_log_start":"154'329909","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3937685823,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":340401,"num_read_kb":1335530,"num_write":337459,"num_write_kb":1349689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"158'327010","reported_seq":648565,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:32.890018+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:26:32.890018+0000","last_peered":"2023-10-14T01:26:32.890018+0000","last_clean":"2023-10-14T01:26:32.890018+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:26:32.890018+0000","last_undegraded":"2023-10-14T01:26:32.890018+0000","last_fullsized":"2023-10-14T01:26:32.890018+0000","mapping_epoch":13,"log_start":"154'317009","ondisk_log_start":"154'317009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3691430157,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":321977,"num_read_kb":1251044,"num_write":317044,"num_write_kb":1267962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"158'332636","reported_seq":662360,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:32.941190+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:26:32.941190+0000","last_peered":"2023-10-14T01:26:32.941190+0000","last_clean":"2023-10-14T01:26:32.941190+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:26:32.941190+0000","last_undegraded":"2023-10-14T01:26:32.941190+0000","last_fullsized":"2023-10-14T01:26:32.941190+0000","mapping_epoch":13,"log_start":"154'322611","ondisk_log_start":"154'322611","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3744772345,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":330174,"num_read_kb":1277644,"num_write":320552,"num_write_kb":1281934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"158'326032","reported_seq":644732,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:35.083834+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:26:35.083834+0000","last_peered":"2023-10-14T01:26:35.083834+0000","last_clean":"2023-10-14T01:26:35.083834+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:26:35.083834+0000","last_undegraded":"2023-10-14T01:26:35.083834+0000","last_fullsized":"2023-10-14T01:26:35.083834+0000","mapping_epoch":13,"log_start":"154'316009","ondisk_log_start":"154'316009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3676713144,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":318969,"num_read_kb":1249447,"num_write":315040,"num_write_kb":1260047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":373,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:14.489692+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:26:14.489692+0000","last_peered":"2023-10-14T01:26:14.489692+0000","last_clean":"2023-10-14T01:26:14.489692+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:26:14.489692+0000","last_undegraded":"2023-10-14T01:26:14.489692+0000","last_fullsized":"2023-10-14T01:26:14.489692+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29610981688,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2576900,"num_read_kb":10055678,"num_write":2539092,"num_write_kb":10154897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20796424192,"data_stored":25107903088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80295,"ondisk_log_size":80295,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739451,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5499648,"kb_used_data":3810424,"kb_used_omap":318,"kb_used_meta":1688897,"kb_avail":88872192,"statfs":{"total":96636764160,"available":91005124608,"internally_reserved":0,"allocated":3901874176,"data_stored":4710963142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1729430534},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:39 2023","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":0.79500000000000004,"15min":0.82399999999999995},"min":{"1min":0.191,"5min":0.187,"15min":0.16300000000000001},"max":{"1min":5.7560000000000002,"5min":12.356,"15min":12.356},"last":0.314},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.53300000000000003,"15min":0.69399999999999995},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":4.6959999999999997,"5min":5.6230000000000002,"15min":7.6680000000000001},"last":0.29399999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:25:39 2023","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.1279999999999999,"15min":1.3320000000000001},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":5.8949999999999996,"5min":8.6159999999999997,"15min":14.005000000000001},"last":10.003},{"interface":"front","average":{"1min":1.577,"5min":1.347,"15min":1.5189999999999999},"min":{"1min":0.28999999999999998,"5min":0.245,"15min":0.20899999999999999},"max":{"1min":7.0570000000000004,"5min":9.9329999999999998,"15min":14.586},"last":10.016}]}]},{"osd":1,"up_from":8,"seq":34359739435,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12360352,"kb_used_data":10157080,"kb_used_omap":824,"kb_used_meta":2202439,"kb_avail":82011488,"statfs":{"total":96636764160,"available":83979763712,"internally_reserved":0,"allocated":10400849920,"data_stored":12556238944,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":844173,"internal_metadata":2255298163},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,11,6,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.85199999999999998,"15min":0.83199999999999996},"min":{"1min":0.24299999999999999,"5min":0.183,"15min":0.183},"max":{"1min":7.3460000000000001,"5min":7.3460000000000001,"15min":23.434000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.97999999999999998,"15min":1.016},"min":{"1min":0.245,"5min":0.221,"15min":0.20899999999999999},"max":{"1min":5.8440000000000003,"5min":8.6460000000000008,"15min":24.908000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:26:06 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1240000000000001,"15min":1.278},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":5.7729999999999997,"5min":6.3310000000000004,"15min":27.837},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.73,"5min":1.3340000000000001,"15min":1.452},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":15.09,"5min":15.09,"15min":25.710000000000001},"last":0.378}]}]},{"osd":0,"up_from":8,"seq":34359739449,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7701880,"kb_used_data":6343852,"kb_used_omap":495,"kb_used_meta":1357520,"kb_avail":86669960,"statfs":{"total":96636764160,"available":88750039040,"internally_reserved":0,"allocated":6496104448,"data_stored":7841521058,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1390100649},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,6,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:25:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":0.99199999999999999,"15min":1.0109999999999999},"min":{"1min":0.222,"5min":0.185,"15min":0.17599999999999999},"max":{"1min":5.3250000000000002,"5min":6.6639999999999997,"15min":11.734999999999999},"last":2.7559999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.76000000000000001,"15min":0.63300000000000001},"min":{"1min":0.23999999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":4.7000000000000002,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.36799999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:25:47 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":0.78600000000000003,"15min":0.873},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.4610000000000003,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":0.312},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.65400000000000003,"15min":0.64500000000000002},"min":{"1min":0.248,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":5.2699999999999996,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.34399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6495580160,"data_stored":7841347194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10399948800,"data_stored":12555688248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3900895232,"data_stored":4710867646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:26:36.808 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:36.808 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:36.809 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:37.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:37.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:37.884 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:37.885+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:38.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:38.597+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:38.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:38.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:38.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:38.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:38.628 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:38.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:38.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:38.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:38.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:38.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:38.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:38.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:41.585 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:41.585+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:26:41.586 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:41.585+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:41.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:41.597+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:41.618 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:41.617+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:41.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:41.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:41.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:41.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:41.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:41.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:42.548 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:42.549+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:42.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:42.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:42.581 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:42.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:42.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:42.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:42.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:42.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:42.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:42.581+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:45.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:26:45.507 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:26:45.510 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:45.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.529+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:45.548 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:45.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:45.549 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.549+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:45.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:45.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:45.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:45.561+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:46.758 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.757+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:46.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:46.800 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:46.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:46.801 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:46.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:46.802 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:46.803 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:46.801+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:47.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:47.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:47.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:47.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:47.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:47.344 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:47.362 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.361+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:47.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:47.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:48.522 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:48.517+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:48.558 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:48.557+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:48.559 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:48.561+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:48.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:48.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:48.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:48.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:48.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:48.565+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:49.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:49.237+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:49.258 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:49.257+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:49.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:49.261+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:49.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:49.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:49.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:49.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:49.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:49.281+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:49.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:49.325+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:49.328 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:49.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:49.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:49.345+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:51.628 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:51.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:26:51.629 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:51.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:51.629 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:51.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:26:51.630 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:51.629+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:51.646 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:51.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:51.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:51.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:26:51.647 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:51.645+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:51.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:51.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:51.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:51.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:51.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:51.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:52.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:52.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:52.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:52.829+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:52.875 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:52.873+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:52.900 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:52.901+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:52.942 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:52.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:52.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:52.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:52.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:52.989+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:53.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:53.217+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:53.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:53.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:53.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:53.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:53.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:53.241+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:53.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:53.273+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:54.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:54.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:54.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:54.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:54.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:54.916+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:54.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:54.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:54.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:54.940+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:54.943 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:54.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:56.810 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:57.018 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:57.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:57.037 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:57.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:57.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:57.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:57.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:57.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:57.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:57.068+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:26:57.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:57.068+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:57.108 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:57.108+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:57.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:57.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:26:57.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:57.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:26:57.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:57.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:26:57.507 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:26:57.508 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:26:57.605 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2879,"stamp":"2023-10-14T01:26:55.933602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29646072664,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2585128,"num_read_kb":10088079,"num_write":2547692,"num_write_kb":10191169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80473,"ondisk_log_size":80473,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24776584,"kb_used_data":20379056,"kb_used_omap":1679,"kb_used_meta":4395824,"kb_avail":258338936,"statfs":{"total":289910292480,"available":264539070464,"internally_reserved":0,"allocated":20868153344,"data_stored":25178047944,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1719639,"internal_metadata":4501324457},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,2,5,14,11,21,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22536192,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5320,"num_read_kb":21212,"num_write":5508,"num_write_kb":22032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007061"},"pg_stats":[{"pgid":"2.7","version":"158'342222","reported_seq":671134,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:55.849016+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:26:55.849016+0000","last_peered":"2023-10-14T01:26:55.849016+0000","last_clean":"2023-10-14T01:26:55.849016+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:26:55.849016+0000","last_undegraded":"2023-10-14T01:26:55.849016+0000","last_fullsized":"2023-10-14T01:26:55.849016+0000","mapping_epoch":13,"log_start":"155'332202","ondisk_log_start":"155'332202","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":883,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 441s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3749900726,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":329416,"num_read_kb":1279923,"num_write":321884,"num_write_kb":1287294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'342266","reported_seq":671623,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:55.822226+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:26:55.822226+0000","last_peered":"2023-10-14T01:26:55.822226+0000","last_clean":"2023-10-14T01:26:55.822226+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:26:55.822226+0000","last_undegraded":"2023-10-14T01:26:55.822226+0000","last_fullsized":"2023-10-14T01:26:55.822226+0000","mapping_epoch":13,"log_start":"155'332213","ondisk_log_start":"155'332213","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3742294384,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":331360,"num_read_kb":1277701,"num_write":321749,"num_write_kb":1286623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'309891","reported_seq":616111,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:55.847296+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:26:55.847296+0000","last_peered":"2023-10-14T01:26:55.847296+0000","last_clean":"2023-10-14T01:26:55.847296+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:26:55.847296+0000","last_undegraded":"2023-10-14T01:26:55.847296+0000","last_fullsized":"2023-10-14T01:26:55.847296+0000","mapping_epoch":13,"log_start":"154'299808","ondisk_log_start":"154'299808","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1190,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 454s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3581907346,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":306414,"num_read_kb":1209468,"num_write":308588,"num_write_kb":1234322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'302201","reported_seq":604187,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:55.784100+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:26:55.784100+0000","last_peered":"2023-10-14T01:26:55.784100+0000","last_clean":"2023-10-14T01:26:55.784100+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:26:55.784100+0000","last_undegraded":"2023-10-14T01:26:55.784100+0000","last_fullsized":"2023-10-14T01:26:55.784100+0000","mapping_epoch":13,"log_start":"154'292107","ondisk_log_start":"154'292107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3503284355,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":302223,"num_read_kb":1191005,"num_write":300925,"num_write_kb":1203622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"158'341113","reported_seq":682307,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.348098+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:26:53.348098+0000","last_peered":"2023-10-14T01:26:53.348098+0000","last_clean":"2023-10-14T01:26:53.348098+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:26:53.348098+0000","last_undegraded":"2023-10-14T01:26:53.348098+0000","last_fullsized":"2023-10-14T01:26:53.348098+0000","mapping_epoch":13,"log_start":"155'331109","ondisk_log_start":"155'331109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3942224191,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":341465,"num_read_kb":1339766,"num_write":338567,"num_write_kb":1354121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"158'328110","reported_seq":650685,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.016527+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:26:53.016527+0000","last_peered":"2023-10-14T01:26:53.016527+0000","last_clean":"2023-10-14T01:26:53.016527+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:26:53.016527+0000","last_undegraded":"2023-10-14T01:26:53.016527+0000","last_fullsized":"2023-10-14T01:26:53.016527+0000","mapping_epoch":13,"log_start":"154'318109","ondisk_log_start":"154'318109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3695845645,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":322979,"num_read_kb":1255040,"num_write":318124,"num_write_kb":1272282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"158'333701","reported_seq":664467,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.042323+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:26:53.042323+0000","last_peered":"2023-10-14T01:26:53.042323+0000","last_clean":"2023-10-14T01:26:53.042323+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:26:53.042323+0000","last_undegraded":"2023-10-14T01:26:53.042323+0000","last_fullsized":"2023-10-14T01:26:53.042323+0000","mapping_epoch":13,"log_start":"154'323611","ondisk_log_start":"154'323611","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3749007609,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":331198,"num_read_kb":1281732,"num_write":321592,"num_write_kb":1286094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"158'327110","reported_seq":646793,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:55.824189+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:26:55.824189+0000","last_peered":"2023-10-14T01:26:55.824189+0000","last_clean":"2023-10-14T01:26:55.824189+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:26:55.824189+0000","last_undegraded":"2023-10-14T01:26:55.824189+0000","last_fullsized":"2023-10-14T01:26:55.824189+0000","mapping_epoch":13,"log_start":"154'317109","ondisk_log_start":"154'317109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3681018040,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":319947,"num_read_kb":1253335,"num_write":316094,"num_write_kb":1264263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":373,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:14.489692+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:26:14.489692+0000","last_peered":"2023-10-14T01:26:14.489692+0000","last_clean":"2023-10-14T01:26:14.489692+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:26:14.489692+0000","last_undegraded":"2023-10-14T01:26:14.489692+0000","last_fullsized":"2023-10-14T01:26:14.489692+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29645482296,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2585002,"num_read_kb":10087970,"num_write":2547523,"num_write_kb":10188621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20865519616,"data_stored":25176998512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80346,"ondisk_log_size":80346,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739456,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5526124,"kb_used_data":3823012,"kb_used_omap":318,"kb_used_meta":1702785,"kb_avail":88845716,"statfs":{"total":96636764160,"available":90978013184,"internally_reserved":0,"allocated":3914764288,"data_stored":4723853254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1743651846},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.76000000000000001,"15min":0.81200000000000006},"min":{"1min":0.189,"5min":0.187,"15min":0.17199999999999999},"max":{"1min":3.5169999999999999,"5min":12.356,"15min":12.356},"last":0.19},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.59799999999999998,"15min":0.65600000000000003},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.191},"max":{"1min":5.3579999999999997,"5min":5.6230000000000002,"15min":7.6680000000000001},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:26:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.262,"15min":1.345},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":11.956,"5min":11.956,"15min":14.005000000000001},"last":0.312},{"interface":"front","average":{"1min":1.585,"5min":1.4239999999999999,"15min":1.496},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":13.342000000000001,"5min":13.342000000000001,"15min":14.586},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739439,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11449356,"kb_used_data":10190724,"kb_used_omap":864,"kb_used_meta":1257759,"kb_avail":82922484,"statfs":{"total":96636764160,"available":84912623616,"internally_reserved":0,"allocated":10435301376,"data_stored":12590690400,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":1287945722},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,3,9,10,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.026,"15min":0.90600000000000003},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.183},"max":{"1min":15.122999999999999,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.1279999999999999,"15min":1.103},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":15.147,"5min":15.147,"15min":24.908000000000001},"last":0.23999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:26:06 2023","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1240000000000001,"15min":1.278},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":5.7729999999999997,"5min":6.3310000000000004,"15min":27.837},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.73,"5min":1.3340000000000001,"15min":1.452},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":15.09,"5min":15.09,"15min":25.710000000000001},"last":0.47999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739453,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7801104,"kb_used_data":6365320,"kb_used_omap":495,"kb_used_meta":1435280,"kb_avail":86570736,"statfs":{"total":96636764160,"available":88648433664,"internally_reserved":0,"allocated":6518087680,"data_stored":7863504290,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1469726889},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,1,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.88900000000000001,"15min":0.98099999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.7559999999999998,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.76700000000000002,"15min":0.63700000000000001},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":5.306,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.23000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.77700000000000002,"15min":0.84899999999999998},"min":{"1min":0.21199999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.3840000000000003,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.65100000000000002,"15min":0.62},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":3.996,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6516879360,"data_stored":7862646394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10434805760,"data_stored":12590545208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3913834496,"data_stored":4723806910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:26:57.607 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:57.607 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:57.607 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:58.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:58.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:26:58.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:58.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:26:58.360 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:58.360+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:26:58.385 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:58.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:26:58.385 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:26:58.380+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:26:58.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:58.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:26:58.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:58.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:26:58.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:58.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:26:58.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:58.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:26:58.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:58.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:26:58.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:26:58.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:00.424 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:00.416+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:00.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:00.428+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:00.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:00.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:00.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:00.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:00.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:00.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:00.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:00.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:00.446 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:00.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:00.927 INFO:tasks.ceph.osd.0.smithi138.stderr:2023-10-14T01:27:00.924+0000 7f5236bbd640 -1 reset not still connected to 0x55c04f89c750 2023-10-14T01:27:03.850 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:03.848+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:04.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:04.208+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:04.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:04.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:04.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:04.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:04.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:04.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:04.254 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:04.252+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:05.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:05.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:05.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:05.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:05.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:05.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:05.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:05.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:05.720 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:05.716+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:05.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:05.748+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:07.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:07.872+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:07.898 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:07.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:07.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:07.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:07.899 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:07.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:09.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:09.736+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:09.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:09.740+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:09.745 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:09.744+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:09.748 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:09.748+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:09.773 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:09.772+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:09.781 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:09.780+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:12.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:12.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:12.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:12.146 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:12.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:12.531 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:12.551 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:12.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:12.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:12.552+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:13.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.496+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:13.512 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.504+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:13.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:13.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:13.522 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.520+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:13.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.528+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:13.894 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.888+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:13.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:13.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:13.897 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.892+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:13.926 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:13.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:14.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:14.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:14.846 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:14.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:14.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:14.844+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:14.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:14.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:14.903 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:14.900+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:14.905 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:14.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:17.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:17.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:17.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:17.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:17.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:17.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:17.187 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:17.188+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:17.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:17.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:17.224 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:17.224+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:17.239 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:17.240+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:17.261 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:17.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:17.269 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:17.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:17.269 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:17.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:17.270 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:17.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:17.608 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:18.142 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:18.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:18.168 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:18.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:18.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:18.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:18.410 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:18.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:18.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:18.411 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.408+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:18.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:18.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:18.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:18.442 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:18.432+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:18.498 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:27:18.499 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:27:18.548 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2890,"stamp":"2023-10-14T01:27:17.942080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29680819032,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2593423,"num_read_kb":10121183,"num_write":2556177,"num_write_kb":10225109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80583,"ondisk_log_size":80583,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25089552,"kb_used_data":20446392,"kb_used_omap":1679,"kb_used_meta":4641456,"kb_avail":258025968,"statfs":{"total":289910292480,"available":264218591232,"internally_reserved":0,"allocated":20937105408,"data_stored":25247000008,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1719639,"internal_metadata":4752851625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,4,15,13,19,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22933504,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5392,"num_read_kb":21512,"num_write":5600,"num_write_kb":22400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004651"},"pg_stats":[{"pgid":"2.7","version":"158'343321","reported_seq":673261,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:16.190734+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:27:16.190734+0000","last_peered":"2023-10-14T01:27:16.190734+0000","last_clean":"2023-10-14T01:27:16.190734+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:27:16.190734+0000","last_undegraded":"2023-10-14T01:27:16.190734+0000","last_fullsized":"2023-10-14T01:27:16.190734+0000","mapping_epoch":13,"log_start":"155'333302","ondisk_log_start":"155'333302","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":912,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 461s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3754140086,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":330422,"num_read_kb":1283939,"num_write":322912,"num_write_kb":1291406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'343458","reported_seq":673916,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:16.179732+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:27:16.179732+0000","last_peered":"2023-10-14T01:27:16.179732+0000","last_clean":"2023-10-14T01:27:16.179732+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:27:16.179732+0000","last_undegraded":"2023-10-14T01:27:16.179732+0000","last_fullsized":"2023-10-14T01:27:16.179732+0000","mapping_epoch":13,"log_start":"155'333413","ondisk_log_start":"155'333413","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3746980208,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":332447,"num_read_kb":1282037,"num_write":322889,"num_write_kb":1291183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'310983","reported_seq":618240,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:16.168154+0000","last_change":"2023-10-14T01:19:21.144562+0000","last_active":"2023-10-14T01:27:16.168154+0000","last_peered":"2023-10-14T01:27:16.168154+0000","last_clean":"2023-10-14T01:27:16.168154+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:27:16.168154+0000","last_undegraded":"2023-10-14T01:27:16.168154+0000","last_fullsized":"2023-10-14T01:27:16.168154+0000","mapping_epoch":13,"log_start":"155'300908","ondisk_log_start":"155'300908","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1232,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 475s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3586273682,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":307442,"num_read_kb":1213552,"num_write":309654,"num_write_kb":1238586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'303185","reported_seq":606144,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:16.156227+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:27:16.156227+0000","last_peered":"2023-10-14T01:27:16.156227+0000","last_clean":"2023-10-14T01:27:16.156227+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:27:16.156227+0000","last_undegraded":"2023-10-14T01:27:16.156227+0000","last_fullsized":"2023-10-14T01:27:16.156227+0000","mapping_epoch":13,"log_start":"154'293107","ondisk_log_start":"154'293107","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3507290243,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":303196,"num_read_kb":1194893,"num_write":301909,"num_write_kb":1207558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"158'342295","reported_seq":684649,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.494863+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:27:13.494863+0000","last_peered":"2023-10-14T01:27:13.494863+0000","last_clean":"2023-10-14T01:27:13.494863+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:27:13.494863+0000","last_undegraded":"2023-10-14T01:27:13.494863+0000","last_fullsized":"2023-10-14T01:27:13.494863+0000","mapping_epoch":13,"log_start":"155'332209","ondisk_log_start":"155'332209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3947061567,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":342616,"num_read_kb":1344358,"num_write":339744,"num_write_kb":1358829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"158'329081","reported_seq":652681,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.177333+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:27:13.177333+0000","last_peered":"2023-10-14T01:27:13.177333+0000","last_clean":"2023-10-14T01:27:13.177333+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:27:13.177333+0000","last_undegraded":"2023-10-14T01:27:13.177333+0000","last_fullsized":"2023-10-14T01:27:13.177333+0000","mapping_epoch":13,"log_start":"155'319009","ondisk_log_start":"155'319009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3699728653,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":323988,"num_read_kb":1259072,"num_write":319075,"num_write_kb":1276086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"158'334844","reported_seq":666664,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.184130+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:27:13.184130+0000","last_peered":"2023-10-14T01:27:13.184130+0000","last_clean":"2023-10-14T01:27:13.184130+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:27:13.184130+0000","last_undegraded":"2023-10-14T01:27:13.184130+0000","last_fullsized":"2023-10-14T01:27:13.184130+0000","mapping_epoch":13,"log_start":"155'324811","ondisk_log_start":"155'324811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3753533689,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":332241,"num_read_kb":1285900,"num_write":322710,"num_write_kb":1290566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"158'328157","reported_seq":648847,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:16.138799+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:27:16.138799+0000","last_peered":"2023-10-14T01:27:16.138799+0000","last_clean":"2023-10-14T01:27:16.138799+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:27:16.138799+0000","last_undegraded":"2023-10-14T01:27:16.138799+0000","last_fullsized":"2023-10-14T01:27:16.138799+0000","mapping_epoch":13,"log_start":"155'318109","ondisk_log_start":"155'318109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3685220536,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":320945,"num_read_kb":1257323,"num_write":317115,"num_write_kb":1268347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":373,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:14.489692+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:26:14.489692+0000","last_peered":"2023-10-14T01:26:14.489692+0000","last_clean":"2023-10-14T01:26:14.489692+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:26:14.489692+0000","last_undegraded":"2023-10-14T01:26:14.489692+0000","last_fullsized":"2023-10-14T01:26:14.489692+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29680228664,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2593297,"num_read_kb":10121074,"num_write":2556008,"num_write_kb":10222561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20934680576,"data_stored":25246159472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80456,"ondisk_log_size":80456,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739460,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5588804,"kb_used_data":3835708,"kb_used_omap":318,"kb_used_meta":1752769,"kb_avail":88783036,"statfs":{"total":96636764160,"available":90913828864,"internally_reserved":0,"allocated":3927764992,"data_stored":4736853958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1794835462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.76000000000000001,"15min":0.81200000000000006},"min":{"1min":0.189,"5min":0.187,"15min":0.17199999999999999},"max":{"1min":3.5169999999999999,"5min":12.356,"15min":12.356},"last":0.189},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.59799999999999998,"15min":0.65600000000000003},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.191},"max":{"1min":5.3579999999999997,"5min":5.6230000000000002,"15min":7.6680000000000001},"last":0.24299999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:26:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.262,"15min":1.345},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":11.956,"5min":11.956,"15min":14.005000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.585,"5min":1.4239999999999999,"15min":1.496},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":13.342000000000001,"5min":13.342000000000001,"15min":14.586},"last":0.29499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739443,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11576844,"kb_used_data":10224580,"kb_used_omap":864,"kb_used_meta":1351391,"kb_avail":82794996,"statfs":{"total":96636764160,"available":84782075904,"internally_reserved":0,"allocated":10469969920,"data_stored":12625358944,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":1383824890},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,9,6,14,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.026,"15min":0.90600000000000003},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.183},"max":{"1min":15.122999999999999,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.1279999999999999,"15min":1.103},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":15.147,"5min":15.147,"15min":24.908000000000001},"last":0.35599999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:27:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1220000000000001,"15min":1.236},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":11.252000000000001,"5min":11.252000000000001,"15min":27.837},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.119,"5min":1.407,"15min":1.379},"min":{"1min":0.253,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":5.7530000000000001,"5min":15.09,"15min":25.710000000000001},"last":0.69699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359739457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7923904,"kb_used_data":6386104,"kb_used_omap":495,"kb_used_meta":1537296,"kb_avail":86447936,"statfs":{"total":96636764160,"available":88522686464,"internally_reserved":0,"allocated":6539370496,"data_stored":7884787106,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1574191273},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.88900000000000001,"15min":0.98099999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.7559999999999998,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.76700000000000002,"15min":0.63700000000000001},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":5.306,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.22800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.77700000000000002,"15min":0.84899999999999998},"min":{"1min":0.21199999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.3840000000000003,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":4.798},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.65100000000000002,"15min":0.62},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":3.996,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.309}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6538547200,"data_stored":7884314234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10469801984,"data_stored":12625541432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3926331392,"data_stored":4736303806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:27:18.550 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:18.550 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:18.551 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:18.876 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:18.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:18.890 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:18.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:18.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:18.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:18.891 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:18.888+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:19.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:19.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:19.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:19.713 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.712+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:19.722 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:19.736 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:19.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:19.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:19.764+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:20.646 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:20.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:20.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:20.628+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:20.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:20.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:21.638 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:21.636+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:21.650 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:21.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:21.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:21.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:21.651 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:21.648+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:22.003 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.004+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:22.021 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.020+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:22.042 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.040+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:22.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:22.059 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.056+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:22.076 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.076+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:22.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.796+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:22.831 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:22.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:22.832 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:22.832+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:24.300 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:24.300+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:24.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:24.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:24.351 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:24.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:24.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:24.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:24.639 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:24.640+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:24.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:24.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:24.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:24.676+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:26.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:26.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:26.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:26.220+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:27.244 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:27.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:27.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:27.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:27.245 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:27.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:27.267 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:27:27.268 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.268+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:27.274 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.272+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:27.285 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:27.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:27.320 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:27.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.320+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:27.353 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:27.348+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:27.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.552+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:27.590 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.588+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:27.605 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.604+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:27.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.604+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:27.606 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:27.604+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:29.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:29.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:29.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:29.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:30.151 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:27:30.152 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:30.154 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.152+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:30.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:30.188 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:30.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:30.189 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.188+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:30.633 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:27:30.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.628+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:30.664 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:30.664+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:30.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:30.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:30.990 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:30.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:30.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:30.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:30.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:30.984+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:30.991 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:30.988+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:30.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:30.992+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:31.218 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:31.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:31.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:31.243 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.240+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:31.864 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.864+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:31.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:31.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:31.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.868+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:31.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.936+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:31.959 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:31.960+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:32.592 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:32.592+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:32.620 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:32.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:32.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:32.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:32.635 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:32.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:32.636 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:32.632+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:33.161 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:33.160+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:33.182 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:33.180+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:33.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:33.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:33.205 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:33.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:33.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:33.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:33.206 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:33.204+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:33.844 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:33.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:33.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:33.856+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:33.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:33.880+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:34.443 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:34.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:34.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:34.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:34.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:34.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:34.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:34.487 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:34.488+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:36.284 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:36.284+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:36.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:36.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:36.309 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:36.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:36.310 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:36.308+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:36.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:36.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:36.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:36.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:36.314 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:36.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:37.804 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:37.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:37.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:37.838 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:37.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.832+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:37.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.840+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:37.854 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.852+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:37.887 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:37.884+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:38.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:38.336+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:38.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:38.368+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:38.372 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:38.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:38.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:38.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:38.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:38.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:38.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:38.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:38.374 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:38.372+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:38.552 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:39.503 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:27:39.504 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:27:39.563 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2900,"stamp":"2023-10-14T01:27:37.948067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29716396888,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2601919,"num_read_kb":10155091,"num_write":2564890,"num_write_kb":10259961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80619,"ondisk_log_size":80619,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25483684,"kb_used_data":20515912,"kb_used_omap":1679,"kb_used_meta":4966064,"kb_avail":257631836,"statfs":{"total":289910292480,"available":263815000064,"internally_reserved":0,"allocated":21008293888,"data_stored":25318188488,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1719639,"internal_metadata":5085250217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,6,19,11,25],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23576576,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5503,"num_read_kb":21952,"num_write":5769,"num_write_kb":23076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004619"},"pg_stats":[{"pgid":"2.7","version":"158'344433","reported_seq":675438,"reported_epoch":158,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:36.569653+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:27:36.569653+0000","last_peered":"2023-10-14T01:27:36.569653+0000","last_clean":"2023-10-14T01:27:36.569653+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:27:36.569653+0000","last_undegraded":"2023-10-14T01:27:36.569653+0000","last_fullsized":"2023-10-14T01:27:36.569653+0000","mapping_epoch":13,"log_start":"155'334402","ondisk_log_start":"155'334402","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":972,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 482s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3758277046,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":331462,"num_read_kb":1288087,"num_write":323925,"num_write_kb":1295458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"158'344656","reported_seq":676228,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:36.543668+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:27:36.543668+0000","last_peered":"2023-10-14T01:27:36.543668+0000","last_clean":"2023-10-14T01:27:36.543668+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:27:36.543668+0000","last_undegraded":"2023-10-14T01:27:36.543668+0000","last_fullsized":"2023-10-14T01:27:36.543668+0000","mapping_epoch":13,"log_start":"155'334613","ondisk_log_start":"155'334613","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3751616880,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":333545,"num_read_kb":1286417,"num_write":324025,"num_write_kb":1295727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"158'312107","reported_seq":620525,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:36.508356+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:27:36.508356+0000","last_peered":"2023-10-14T01:27:36.508356+0000","last_clean":"2023-10-14T01:27:36.508356+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:27:36.508356+0000","last_undegraded":"2023-10-14T01:27:36.508356+0000","last_fullsized":"2023-10-14T01:27:36.508356+0000","mapping_epoch":13,"log_start":"155'302008","ondisk_log_start":"155'302008","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3590816146,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":308598,"num_read_kb":1218168,"num_write":310768,"num_write_kb":1243042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"158'304262","reported_seq":608210,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:36.516401+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:27:36.516401+0000","last_peered":"2023-10-14T01:27:36.516401+0000","last_clean":"2023-10-14T01:27:36.516401+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:27:36.516401+0000","last_undegraded":"2023-10-14T01:27:36.516401+0000","last_fullsized":"2023-10-14T01:27:36.516401+0000","mapping_epoch":13,"log_start":"155'294207","ondisk_log_start":"155'294207","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3511693443,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":304185,"num_read_kb":1198841,"num_write":302986,"num_write_kb":1211866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"158'343394","reported_seq":686888,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:33.613422+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:27:33.613422+0000","last_peered":"2023-10-14T01:27:33.613422+0000","last_clean":"2023-10-14T01:27:33.613422+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:27:33.613422+0000","last_undegraded":"2023-10-14T01:27:33.613422+0000","last_fullsized":"2023-10-14T01:27:33.613422+0000","mapping_epoch":13,"log_start":"155'333309","ondisk_log_start":"155'333309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3951534399,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":343752,"num_read_kb":1348898,"num_write":340839,"num_write_kb":1363209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"158'330279","reported_seq":654968,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:33.414402+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:27:33.414402+0000","last_peered":"2023-10-14T01:27:33.414402+0000","last_clean":"2023-10-14T01:27:33.414402+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:27:33.414402+0000","last_undegraded":"2023-10-14T01:27:33.414402+0000","last_fullsized":"2023-10-14T01:27:33.414402+0000","mapping_epoch":13,"log_start":"155'320209","ondisk_log_start":"155'320209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3704447245,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":325064,"num_read_kb":1263368,"num_write":320221,"num_write_kb":1280670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"158'336062","reported_seq":668993,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:33.409978+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:27:33.409978+0000","last_peered":"2023-10-14T01:27:33.409978+0000","last_clean":"2023-10-14T01:27:33.409978+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:27:33.409978+0000","last_undegraded":"2023-10-14T01:27:33.409978+0000","last_fullsized":"2023-10-14T01:27:33.409978+0000","mapping_epoch":13,"log_start":"155'326011","ondisk_log_start":"155'326011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3758215417,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":333324,"num_read_kb":1290220,"num_write":323863,"num_write_kb":1295178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"158'329167","reported_seq":650783,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:27:36.472589+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:27:36.472589+0000","last_peered":"2023-10-14T01:27:36.472589+0000","last_clean":"2023-10-14T01:27:36.472589+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:27:36.472589+0000","last_undegraded":"2023-10-14T01:27:36.472589+0000","last_fullsized":"2023-10-14T01:27:36.472589+0000","mapping_epoch":13,"log_start":"155'319109","ondisk_log_start":"155'319109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3689205944,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":321863,"num_read_kb":1260983,"num_write":318094,"num_write_kb":1272263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":373,"reported_epoch":157,"state":"active+clean","last_fresh":"2023-10-14T01:26:14.489692+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:26:14.489692+0000","last_peered":"2023-10-14T01:26:14.489692+0000","last_clean":"2023-10-14T01:26:14.489692+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:26:14.489692+0000","last_undegraded":"2023-10-14T01:26:14.489692+0000","last_fullsized":"2023-10-14T01:26:14.489692+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29715806520,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2601793,"num_read_kb":10154982,"num_write":2564721,"num_write_kb":10257413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21006032896,"data_stored":25317511792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80492,"ondisk_log_size":80492,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739464,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5669992,"kb_used_data":3848864,"kb_used_omap":318,"kb_used_meta":1820801,"kb_avail":88701848,"statfs":{"total":96636764160,"available":90830692352,"internally_reserved":0,"allocated":3941236736,"data_stored":4750325702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1864500230},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,3,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.76000000000000001,"15min":0.81200000000000006},"min":{"1min":0.189,"5min":0.187,"15min":0.17199999999999999},"max":{"1min":3.5169999999999999,"5min":12.356,"15min":12.356},"last":0.219},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.59799999999999998,"15min":0.65600000000000003},"min":{"1min":0.22500000000000001,"5min":0.191,"15min":0.191},"max":{"1min":5.3579999999999997,"5min":5.6230000000000002,"15min":7.6680000000000001},"last":0.29699999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:26:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.262,"15min":1.345},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":11.956,"5min":11.956,"15min":14.005000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.585,"5min":1.4239999999999999,"15min":1.496},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":13.342000000000001,"5min":13.342000000000001,"15min":14.586},"last":0.378}]}]},{"osd":1,"up_from":8,"seq":34359739447,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11792156,"kb_used_data":10259664,"kb_used_omap":864,"kb_used_meta":1531615,"kb_avail":82579684,"statfs":{"total":96636764160,"available":84561596416,"internally_reserved":0,"allocated":10505895936,"data_stored":12661284960,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":1568374266},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,12,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":1.026,"15min":0.90600000000000003},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.183},"max":{"1min":15.122999999999999,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.7529999999999999,"5min":1.1279999999999999,"15min":1.103},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":15.147,"5min":15.147,"15min":24.908000000000001},"last":2.5529999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:27:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1220000000000001,"15min":1.236},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":11.252000000000001,"5min":11.252000000000001,"15min":27.837},"last":2.7010000000000001},{"interface":"front","average":{"1min":1.119,"5min":1.407,"15min":1.379},"min":{"1min":0.253,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":5.7530000000000001,"5min":15.09,"15min":25.710000000000001},"last":2.5960000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739461,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8021536,"kb_used_data":6407384,"kb_used_omap":495,"kb_used_meta":1613648,"kb_avail":86350304,"statfs":{"total":96636764160,"available":88422711296,"internally_reserved":0,"allocated":6561161216,"data_stored":7906577826,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1652375721},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.88900000000000001,"15min":0.98099999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.7559999999999998,"5min":6.6639999999999997,"15min":11.734999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.76700000000000002,"15min":0.63700000000000001},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":5.306,"5min":6.5110000000000001,"15min":6.5110000000000001},"last":0.36899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.77700000000000002,"15min":0.84899999999999998},"min":{"1min":0.21199999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.3840000000000003,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":1.79},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.65100000000000002,"15min":0.62},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":3.996,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6560546816,"data_stored":7906313850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10505654272,"data_stored":12661393720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3939831808,"data_stored":4749804222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:27:39.565 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:39.565 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:41.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:27:41.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:27:41.746 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:41.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:41.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:41.747 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:41.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:41.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:41.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:41.761 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:41.760+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:44.002 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.000+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:44.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:44.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:44.016 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.016+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:44.049 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.048+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:44.055 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:44.061 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.060+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:44.122 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.120+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:44.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:44.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:44.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:44.895 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:44.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:44.902 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:44.900+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:44.909 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:44.908+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:44.923 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:44.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:44.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:44.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:44.924 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:44.920+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:46.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:46.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:46.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:46.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:46.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.192+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:46.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:46.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:46.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:46.262 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.260+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:46.271 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:46.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:46.272 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.272+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:46.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:46.295 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:46.296 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:46.292+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:47.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:47.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:47.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:47.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:47.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:47.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:47.165 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:47.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:47.175 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:47.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:47.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:47.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:47.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:47.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:47.857 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:47.856+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:47.876 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:47.876+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:47.897 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:47.896+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:47.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:47.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:47.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:47.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:47.906 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:47.904+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:48.847 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:48.843+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:48.860 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:48.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:48.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:48.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:48.861 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:48.855+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:49.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:49.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:49.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:49.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:50.039 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:50.035+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:50.046 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:50.043+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:50.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:50.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:50.999 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:50.995+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:51.010 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:51.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:51.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:51.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:51.011 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:51.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:51.029 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:51.023+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:51.087 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.083+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:51.154 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.151+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:51.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.167+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:51.174 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.171+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:51.395 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:51.391+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:51.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:51.395+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:51.758 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.755+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:51.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.827+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:51.849 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:51.850 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:51.850 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:51.851 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:51.847+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:52.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:52.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:52.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:52.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:52.545 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:52.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:52.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:52.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:53.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.047+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:53.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:53.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:53.080 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.075+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:53.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.087+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:53.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.091+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:53.127 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:53.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:53.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:27:53.540 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:53.531+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:27:53.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.539+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:27:53.547 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.543+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:53.554 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:53.551+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:53.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:53.560 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.559+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:53.565 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:53.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:27:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:53.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:53.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:53.566 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:53.563+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:53.573 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.567+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:53.576 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:53.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:53.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:56.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.147+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:56.162 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.159+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:56.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:56.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.167+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:56.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.175+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:56.343 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:27:56.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:56.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:56.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:56.370 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.367+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:56.428 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.427+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:56.445 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.443+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:56.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.671+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:56.708 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:27:56.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:56.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.703+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:56.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:56.723 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:56.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:57.150 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:57.147+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:27:57.166 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:57.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:27:57.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:57.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:27:57.167 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:27:57.163+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:27:57.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:57.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:27:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:57.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:27:57.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:57.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:27:57.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:57.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:57.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:57.847+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:58.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:58.471+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:27:58.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:58.483+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:27:58.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:27:58.503+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:27:59.566 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:00.075 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:00.071+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:00.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:00.083+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:00.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:00.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:00.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:00.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:00.111 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:00.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:00.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:00.135+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:00.426 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:28:00.427 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:28:00.525 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2912,"stamp":"2023-10-14T01:27:59.960671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29756373848,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2611971,"num_read_kb":10195207,"num_write":2574666,"num_write_kb":10299065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80572,"ondisk_log_size":80572,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25862944,"kb_used_data":20597764,"kb_used_omap":1679,"kb_used_meta":5263472,"kb_avail":257252576,"statfs":{"total":289910292480,"available":263426637824,"internally_reserved":0,"allocated":21092110336,"data_stored":25401991691,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1719639,"internal_metadata":5389796009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,9,14,16,17,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21803008,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5455,"num_read_kb":21756,"num_write":5322,"num_write_kb":21288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007343"},"pg_stats":[{"pgid":"2.7","version":"159'345549","reported_seq":677673,"reported_epoch":159,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:57.049997+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:27:57.049997+0000","last_peered":"2023-10-14T01:27:57.049997+0000","last_clean":"2023-10-14T01:27:57.049997+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:27:57.049997+0000","last_undegraded":"2023-10-14T01:27:57.049997+0000","last_fullsized":"2023-10-14T01:27:57.049997+0000","mapping_epoch":13,"log_start":"155'335503","ondisk_log_start":"155'335503","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1005,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 502s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3762483638,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":332558,"num_read_kb":1292467,"num_write":324952,"num_write_kb":1299566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'345842","reported_seq":678569,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:57.009980+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:27:57.009980+0000","last_peered":"2023-10-14T01:27:57.009980+0000","last_clean":"2023-10-14T01:27:57.009980+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:27:57.009980+0000","last_undegraded":"2023-10-14T01:27:57.009980+0000","last_fullsized":"2023-10-14T01:27:57.009980+0000","mapping_epoch":13,"log_start":"155'335813","ondisk_log_start":"155'335813","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3756196208,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":334681,"num_read_kb":1290957,"num_write":325142,"num_write_kb":1300195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'313251","reported_seq":622827,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:57.039404+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:27:57.039404+0000","last_peered":"2023-10-14T01:27:57.039404+0000","last_clean":"2023-10-14T01:27:57.039404+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:27:57.039404+0000","last_undegraded":"2023-10-14T01:27:57.039404+0000","last_fullsized":"2023-10-14T01:27:57.039404+0000","mapping_epoch":13,"log_start":"155'303208","ondisk_log_start":"155'303208","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3595477394,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":309752,"num_read_kb":1222772,"num_write":311912,"num_write_kb":1247618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'305424","reported_seq":610590,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:57.023341+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:27:57.023341+0000","last_peered":"2023-10-14T01:27:57.023341+0000","last_clean":"2023-10-14T01:27:57.023341+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:27:57.023341+0000","last_undegraded":"2023-10-14T01:27:57.023341+0000","last_fullsized":"2023-10-14T01:27:57.023341+0000","mapping_epoch":13,"log_start":"155'295407","ondisk_log_start":"155'295407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3516436611,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":305402,"num_read_kb":1203693,"num_write":304148,"num_write_kb":1216514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"159'344897","reported_seq":689870,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:58.908204+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:27:58.908204+0000","last_peered":"2023-10-14T01:27:58.908204+0000","last_clean":"2023-10-14T01:27:58.908204+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:27:58.908204+0000","last_undegraded":"2023-10-14T01:27:58.908204+0000","last_fullsized":"2023-10-14T01:27:58.908204+0000","mapping_epoch":13,"log_start":"155'334809","ondisk_log_start":"155'334809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3957711167,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":345227,"num_read_kb":1354798,"num_write":342337,"num_write_kb":1369201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"159'331696","reported_seq":657842,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:58.647315+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:27:58.647315+0000","last_peered":"2023-10-14T01:27:58.647315+0000","last_clean":"2023-10-14T01:27:58.647315+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:27:58.647315+0000","last_undegraded":"2023-10-14T01:27:58.647315+0000","last_fullsized":"2023-10-14T01:27:58.647315+0000","mapping_epoch":13,"log_start":"155'321609","ondisk_log_start":"155'321609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3710075149,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":326498,"num_read_kb":1269096,"num_write":321602,"num_write_kb":1286194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"159'337450","reported_seq":671767,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:58.614080+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:27:58.614080+0000","last_peered":"2023-10-14T01:27:58.614080+0000","last_clean":"2023-10-14T01:27:58.614080+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:27:58.614080+0000","last_undegraded":"2023-10-14T01:27:58.614080+0000","last_fullsized":"2023-10-14T01:27:58.614080+0000","mapping_epoch":13,"log_start":"155'327411","ondisk_log_start":"155'327411","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3763704057,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":334681,"num_read_kb":1295624,"num_write":325206,"num_write_kb":1300550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"159'330305","reported_seq":653111,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:57.029278+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:27:57.029278+0000","last_peered":"2023-10-14T01:27:57.029278+0000","last_clean":"2023-10-14T01:27:57.029278+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:27:57.029278+0000","last_undegraded":"2023-10-14T01:27:57.029278+0000","last_fullsized":"2023-10-14T01:27:57.029278+0000","mapping_epoch":13,"log_start":"155'320209","ondisk_log_start":"155'320209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3693699256,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":323046,"num_read_kb":1265691,"num_write":319198,"num_write_kb":1276679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":374,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-10-14T01:26:29.922723+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:26:29.922723+0000","last_peered":"2023-10-14T01:26:29.922723+0000","last_clean":"2023-10-14T01:26:29.922723+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:26:29.922723+0000","last_undegraded":"2023-10-14T01:26:29.922723+0000","last_fullsized":"2023-10-14T01:26:29.922723+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29755783480,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2611845,"num_read_kb":10195098,"num_write":2574497,"num_write_kb":10296517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21088096256,"data_stored":25399575152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80445,"ondisk_log_size":80445,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739469,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5740128,"kb_used_data":3864856,"kb_used_omap":318,"kb_used_meta":1874945,"kb_avail":88631712,"statfs":{"total":96636764160,"available":90758873088,"internally_reserved":0,"allocated":3957612544,"data_stored":4766697095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1919943686},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,2,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.79100000000000004,"15min":0.82499999999999996},"min":{"1min":0.189,"5min":0.189,"15min":0.17199999999999999},"max":{"1min":4.0279999999999996,"5min":12.356,"15min":12.356},"last":0.441},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.52000000000000002,"15min":0.62},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.191},"max":{"1min":1.343,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.38900000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.2649999999999999,"15min":1.282},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":3.8929999999999998,"5min":11.956,"15min":14.005000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.333,"15min":1.4079999999999999},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.7639999999999998,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.58999999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739451,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11965216,"kb_used_data":10296724,"kb_used_omap":864,"kb_used_meta":1667615,"kb_avail":82406624,"statfs":{"total":96636764160,"available":84384382976,"internally_reserved":0,"allocated":10543845376,"data_stored":12699229985,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":1707638266},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,11,8,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":1.0369999999999999,"15min":0.90800000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.183},"max":{"1min":3.9710000000000001,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.752,"5min":1.175,"15min":1.1180000000000001},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":5.3220000000000001,"5min":15.147,"15min":24.908000000000001},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:27:09 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.1220000000000001,"15min":1.236},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":11.252000000000001,"5min":11.252000000000001,"15min":27.837},"last":0.315},{"interface":"front","average":{"1min":1.119,"5min":1.407,"15min":1.379},"min":{"1min":0.253,"5min":0.22,"15min":0.16800000000000001},"max":{"1min":5.7530000000000001,"5min":15.09,"15min":25.710000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739466,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8157600,"kb_used_data":6436184,"kb_used_omap":495,"kb_used_meta":1720912,"kb_avail":86214240,"statfs":{"total":96636764160,"available":88283381760,"internally_reserved":0,"allocated":6590652416,"data_stored":7936064611,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1762214057},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.76500000000000001,"15min":0.93999999999999995},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.1950000000000003,"5min":5.3250000000000002,"15min":11.734999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.67200000000000004,"15min":0.63900000000000001},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":2.0379999999999998,"5min":5.7619999999999996,"15min":6.5110000000000001},"last":0.38400000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:27:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.82799999999999996,"15min":0.876},"min":{"1min":0.23300000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":4.798,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66400000000000003,"15min":0.63100000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":4.024,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6589112320,"data_stored":7934879354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10542473216,"data_stored":12698212664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3956510720,"data_stored":4766483134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:28:00.526 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:00.527 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:02.220 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.219+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:02.228 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:02.229 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:02.230 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.223+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:02.280 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:02.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.319+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:02.361 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.359+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:02.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.583+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:02.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:02.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:02.938 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:02.935+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:02.968 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:02.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:02.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:02.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:02.969 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:02.967+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:05.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.195+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:05.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:05.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.203+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:05.239 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.239+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:05.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.279+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:05.466 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.459+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:05.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:05.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:05.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:05.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.507+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:05.539 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:05.820 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:05.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:05.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.819+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:05.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:05.831+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:06.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:06.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:06.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:06.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:06.613 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:06.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:06.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:06.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:06.636 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:06.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:06.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:06.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:06.637 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:06.627+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:07.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:07.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:07.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:07.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:07.375 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:07.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:07.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:07.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:07.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:07.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:07.386 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:07.383+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:10.107 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:10.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:10.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:10.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:10.108 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:10.107+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:10.118 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:10.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:10.140 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:10.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:10.147 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:10.143+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:12.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:12.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:12.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:12.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:13.965 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:13.963+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:14.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:14.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.007+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:14.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:14.022 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:14.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.019+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:14.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:14.885 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:14.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.883+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:14.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.939+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:14.951 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.951+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:14.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:14.971+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:15.774 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:15.771+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:15.784 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:15.783+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:15.790 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:15.787+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:15.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:15.799+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:15.803 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:15.799+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:15.804 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:15.799+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:16.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:16.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:16.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:16.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:16.452 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:16.451+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:16.467 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:16.463+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:17.358 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:17.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:17.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:17.359 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:17.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:17.360 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.355+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:17.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:17.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:17.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:17.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:17.701 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.699+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:17.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:17.707+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:17.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:17.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:28:17.743 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:17.739+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:17.775 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:17.763+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:17.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:17.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:17.777 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:17.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:17.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:17.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:17.778 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:17.775+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:18.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:18.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:18.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:18.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.191+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:18.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:18.203 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.199+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:18.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:18.356 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:18.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:18.364 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:18.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:18.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.375+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:18.419 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:18.411+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:19.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:19.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:19.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:19.315+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:19.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:19.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:19.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:19.347+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:20.528 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:21.280 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:28:21.281 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:28:21.337 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2922,"stamp":"2023-10-14T01:28:19.965662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29793061720,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2620843,"num_read_kb":10230587,"num_write":2583649,"num_write_kb":10334997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80420,"ondisk_log_size":80420,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":26223064,"kb_used_data":20669244,"kb_used_omap":1679,"kb_used_meta":5552112,"kb_avail":256892456,"statfs":{"total":289910292480,"available":263057874944,"internally_reserved":0,"allocated":21165305856,"data_stored":25475187211,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1719639,"internal_metadata":5685363369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,3,8,16,8,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22097920,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5243,"num_read_kb":20908,"num_write":5433,"num_write_kb":21732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"2.7","version":"159'346870","reported_seq":680236,"reported_epoch":159,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:17.490112+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:28:17.490112+0000","last_peered":"2023-10-14T01:28:17.490112+0000","last_clean":"2023-10-14T01:28:17.490112+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:28:17.490112+0000","last_undegraded":"2023-10-14T01:28:17.490112+0000","last_fullsized":"2023-10-14T01:28:17.490112+0000","mapping_epoch":13,"log_start":"156'336803","ondisk_log_start":"156'336803","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1035,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 523s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3767497142,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":333779,"num_read_kb":1297331,"num_write":326178,"num_write_kb":1304470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'347087","reported_seq":680963,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.487404+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:28:17.487404+0000","last_peered":"2023-10-14T01:28:17.487404+0000","last_clean":"2023-10-14T01:28:17.487404+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:28:17.487404+0000","last_undegraded":"2023-10-14T01:28:17.487404+0000","last_fullsized":"2023-10-14T01:28:17.487404+0000","mapping_epoch":13,"log_start":"157'337013","ondisk_log_start":"157'337013","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3760976240,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":335813,"num_read_kb":1295481,"num_write":326308,"num_write_kb":1304859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'314315","reported_seq":624936,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.488433+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:28:17.488433+0000","last_peered":"2023-10-14T01:28:17.488433+0000","last_clean":"2023-10-14T01:28:17.488433+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:28:17.488433+0000","last_undegraded":"2023-10-14T01:28:17.488433+0000","last_fullsized":"2023-10-14T01:28:17.488433+0000","mapping_epoch":13,"log_start":"156'304308","ondisk_log_start":"156'304308","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3599827346,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":310796,"num_read_kb":1226932,"num_write":312976,"num_write_kb":1251874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'306430","reported_seq":612603,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.486145+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:28:17.486145+0000","last_peered":"2023-10-14T01:28:17.486145+0000","last_clean":"2023-10-14T01:28:17.486145+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:28:17.486145+0000","last_undegraded":"2023-10-14T01:28:17.486145+0000","last_fullsized":"2023-10-14T01:28:17.486145+0000","mapping_epoch":13,"log_start":"155'296407","ondisk_log_start":"155'296407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3520520323,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":306408,"num_read_kb":1207713,"num_write":305154,"num_write_kb":1220538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"159'346136","reported_seq":692293,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:19.159685+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:28:19.159685+0000","last_peered":"2023-10-14T01:28:19.159685+0000","last_clean":"2023-10-14T01:28:19.159685+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:28:19.159685+0000","last_undegraded":"2023-10-14T01:28:19.159685+0000","last_fullsized":"2023-10-14T01:28:19.159685+0000","mapping_epoch":13,"log_start":"157'336109","ondisk_log_start":"157'336109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3962704191,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":346407,"num_read_kb":1359502,"num_write":343572,"num_write_kb":1374141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"159'332865","reported_seq":660183,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:18.720559+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:28:18.720559+0000","last_peered":"2023-10-14T01:28:18.720559+0000","last_clean":"2023-10-14T01:28:18.720559+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:28:18.720559+0000","last_undegraded":"2023-10-14T01:28:18.720559+0000","last_fullsized":"2023-10-14T01:28:18.720559+0000","mapping_epoch":13,"log_start":"156'322809","ondisk_log_start":"156'322809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3714777357,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":327655,"num_read_kb":1273712,"num_write":322751,"num_write_kb":1290790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"159'338547","reported_seq":673977,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:18.665838+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:28:18.665838+0000","last_peered":"2023-10-14T01:28:18.665838+0000","last_clean":"2023-10-14T01:28:18.665838+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:28:18.665838+0000","last_undegraded":"2023-10-14T01:28:18.665838+0000","last_fullsized":"2023-10-14T01:28:18.665838+0000","mapping_epoch":13,"log_start":"156'328511","ondisk_log_start":"156'328511","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3768049913,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":335774,"num_read_kb":1299968,"num_write":326278,"num_write_kb":1304838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"159'331412","reported_seq":655267,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.443330+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:28:17.443330+0000","last_peered":"2023-10-14T01:28:17.443330+0000","last_clean":"2023-10-14T01:28:17.443330+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:28:17.443330+0000","last_undegraded":"2023-10-14T01:28:17.443330+0000","last_fullsized":"2023-10-14T01:28:17.443330+0000","mapping_epoch":13,"log_start":"156'321409","ondisk_log_start":"156'321409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3698118840,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":324085,"num_read_kb":1269839,"num_write":320263,"num_write_kb":1280939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":375,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:44.961218+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:27:44.961218+0000","last_peered":"2023-10-14T01:27:44.961218+0000","last_clean":"2023-10-14T01:27:44.961218+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:27:44.961218+0000","last_undegraded":"2023-10-14T01:27:44.961218+0000","last_fullsized":"2023-10-14T01:27:44.961218+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29792471352,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2620717,"num_read_kb":10230478,"num_write":2583480,"num_write_kb":10332449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21161652224,"data_stored":25473131120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80293,"ondisk_log_size":80293,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739473,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5830720,"kb_used_data":3877816,"kb_used_omap":318,"kb_used_meta":1952577,"kb_avail":88541120,"statfs":{"total":96636764160,"available":90666106880,"internally_reserved":0,"allocated":3970883584,"data_stored":4779968135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":326650,"internal_metadata":1999438854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.79100000000000004,"15min":0.82499999999999996},"min":{"1min":0.189,"5min":0.189,"15min":0.17199999999999999},"max":{"1min":4.0279999999999996,"5min":12.356,"15min":12.356},"last":1.8220000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.52000000000000002,"15min":0.62},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.191},"max":{"1min":1.343,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.2649999999999999,"15min":1.282},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":3.8929999999999998,"5min":11.956,"15min":14.005000000000001},"last":0.621},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.333,"15min":1.4079999999999999},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.7639999999999998,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.88900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739455,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12135992,"kb_used_data":10332848,"kb_used_omap":864,"kb_used_meta":1802271,"kb_avail":82235848,"statfs":{"total":96636764160,"available":84209508352,"internally_reserved":0,"allocated":10580836352,"data_stored":12736220961,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":1845526010},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,6,15,6,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":1.0369999999999999,"15min":0.90800000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.183},"max":{"1min":3.9710000000000001,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.752,"5min":1.175,"15min":1.1180000000000001},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":5.3220000000000001,"5min":15.147,"15min":24.908000000000001},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:28:10 2023","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.298,"15min":1.3109999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23499999999999999},"max":{"1min":12.818,"5min":12.818,"15min":27.837},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.327,"15min":1.353},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":9.7669999999999995,"5min":15.09,"15min":25.710000000000001},"last":0.432}]}]},{"osd":0,"up_from":8,"seq":34359739470,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8256352,"kb_used_data":6458580,"kb_used_omap":495,"kb_used_meta":1797264,"kb_avail":86115488,"statfs":{"total":96636764160,"available":88182259712,"internally_reserved":0,"allocated":6613585920,"data_stored":7958998115,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1840398505},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.76500000000000001,"15min":0.93999999999999995},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.1950000000000003,"5min":5.3250000000000002,"15min":11.734999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.67200000000000004,"15min":0.63900000000000001},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":2.0379999999999998,"5min":5.7619999999999996,"15min":6.5110000000000001},"last":0.59199999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:27:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.82799999999999996,"15min":0.876},"min":{"1min":0.23300000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":4.798,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66400000000000003,"15min":0.63100000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":4.024,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.22700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6612361216,"data_stored":7958128250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10579431424,"data_stored":12735170872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3969859584,"data_stored":4779831998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:28:21.339 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:21.339 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:21.377 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:21.375+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:21.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:21.419+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:21.437 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:21.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:21.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:21.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:21.438 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:21.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:21.439 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:21.435+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:22.642 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.639+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:22.653 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.651+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:22.666 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.663+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:22.673 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.671+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:22.673 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.671+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:22.674 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.671+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:22.806 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.803+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:22.828 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.827+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:22.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.839+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:22.840 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.839+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:22.840 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:22.839+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:23.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.455+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:28:23.544 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.535+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:23.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:23.561 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:23.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.555+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:23.578 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.575+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:23.597 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.595+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:23.601 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.599+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:23.676 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:23.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:23.697 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.695+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:23.727 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:23.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.727+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:23.733 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:23.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:25.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:25.295+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:25.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:25.303+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:26.874 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:26.871+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:26.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:26.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:26.911 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:26.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:26.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:26.903+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:26.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:26.919+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:26.929 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:26.927+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:26.975 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:26.975+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:28.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:28:28.028 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.027+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:28.067 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:28.068 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.063+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:28.078 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.075+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:28.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.075+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:28.079 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.075+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:28.340 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:28.339+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:28.351 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:28.351+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:28.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:28.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:28.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:28.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:28.369 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:28.363+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:28.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:28.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:28.381 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:28.379+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:28.555 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.555+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:28:28.556 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.555+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:28.570 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.567+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:28.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.579+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:28.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.579+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:28.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.579+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:28.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:28.579+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:31.117 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.111+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:31.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:31.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:31.125 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.123+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:31.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.139+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:31.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:28:31.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.579+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:28:31.593 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.587+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:31.595 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.591+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:31.619 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:31.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:31.620 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.615+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:31.631 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.631+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:31.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:31.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:31.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:32.119 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:32.115+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:32.155 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:32.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:32.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:32.155+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:32.164 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:32.163+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:32.595 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:32.595+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:32.600 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:32.599+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:32.626 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:32.619+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:32.627 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:32.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:32.627 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:32.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:32.628 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:32.623+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:33.151 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:33.151+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:33.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:33.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:33.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:33.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:33.183 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:33.179+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:33.184 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:33.183+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:34.695 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:34.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:34.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:34.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:34.696 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:34.691+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:34.740 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:34.731+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:34.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:34.795+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:34.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:34.807+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:38.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:38.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:38.661 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:38.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:38.662 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.659+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:38.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:38.969 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:38.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:38.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:38.970 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:38.971 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.967+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:38.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:38.991+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:40.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.639+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:28:40.648 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.647+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:40.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:40.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:40.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:40.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.683+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:40.714 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.711+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:40.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.715+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:40.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:40.719+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:41.340 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:41.647 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:41.643+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:41.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:41.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:41.656 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:41.655+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:41.663 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:41.663+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:41.669 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:41.667+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:41.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:41.675+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:42.061 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:28:42.062 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:28:42.127 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2933,"stamp":"2023-10-14T01:28:41.978781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29829761880,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2629697,"num_read_kb":10265943,"num_write":2592659,"num_write_kb":10371037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80566,"ondisk_log_size":80566,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25285744,"kb_used_data":20740312,"kb_used_omap":1696,"kb_used_meta":4543711,"kb_avail":257829776,"statfs":{"total":289910292480,"available":264017690624,"internally_reserved":0,"allocated":21238079488,"data_stored":25547956747,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1737094,"internal_metadata":4652760698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,3,12,13,6,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18685952,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4409,"num_read_kb":17620,"num_write":4591,"num_write_kb":18364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004782"},"pg_stats":[{"pgid":"2.7","version":"159'348045","reported_seq":682567,"reported_epoch":159,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:37.997262+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:28:37.997262+0000","last_peered":"2023-10-14T01:28:37.997262+0000","last_clean":"2023-10-14T01:28:37.997262+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:28:37.997262+0000","last_undegraded":"2023-10-14T01:28:37.997262+0000","last_fullsized":"2023-10-14T01:28:37.997262+0000","mapping_epoch":13,"log_start":"157'338004","ondisk_log_start":"157'338004","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1074,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 543s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3771916726,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":334917,"num_read_kb":1301867,"num_write":327265,"num_write_kb":1308818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"159'348447","reported_seq":683522,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:37.956171+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:28:37.956171+0000","last_peered":"2023-10-14T01:28:37.956171+0000","last_clean":"2023-10-14T01:28:37.956171+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:28:37.956171+0000","last_undegraded":"2023-10-14T01:28:37.956171+0000","last_fullsized":"2023-10-14T01:28:37.956171+0000","mapping_epoch":13,"log_start":"157'338413","ondisk_log_start":"157'338413","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3766272368,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":337010,"num_read_kb":1300269,"num_write":327608,"num_write_kb":1310059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"159'315372","reported_seq":627026,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:37.996025+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:28:37.996025+0000","last_peered":"2023-10-14T01:28:37.996025+0000","last_clean":"2023-10-14T01:28:37.996025+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:28:37.996025+0000","last_undegraded":"2023-10-14T01:28:37.996025+0000","last_fullsized":"2023-10-14T01:28:37.996025+0000","mapping_epoch":13,"log_start":"157'305308","ondisk_log_start":"157'305308","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3604148626,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":311830,"num_read_kb":1231060,"num_write":314033,"num_write_kb":1256102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"159'307471","reported_seq":614706,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:37.987062+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:28:37.987062+0000","last_peered":"2023-10-14T01:28:37.987062+0000","last_clean":"2023-10-14T01:28:37.987062+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:28:37.987062+0000","last_undegraded":"2023-10-14T01:28:37.987062+0000","last_fullsized":"2023-10-14T01:28:37.987062+0000","mapping_epoch":13,"log_start":"156'297407","ondisk_log_start":"156'297407","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3524747395,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":307477,"num_read_kb":1211977,"num_write":306195,"num_write_kb":1224702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"159'347258","reported_seq":694481,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:39.410394+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:28:39.410394+0000","last_peered":"2023-10-14T01:28:39.410394+0000","last_clean":"2023-10-14T01:28:39.410394+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:28:39.410394+0000","last_undegraded":"2023-10-14T01:28:39.410394+0000","last_fullsized":"2023-10-14T01:28:39.410394+0000","mapping_epoch":13,"log_start":"157'337209","ondisk_log_start":"157'337209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3967238463,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":347470,"num_read_kb":1363750,"num_write":344689,"num_write_kb":1378609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"159'334031","reported_seq":662437,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:38.912127+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:28:38.912127+0000","last_peered":"2023-10-14T01:28:38.912127+0000","last_clean":"2023-10-14T01:28:38.912127+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:28:38.912127+0000","last_undegraded":"2023-10-14T01:28:38.912127+0000","last_fullsized":"2023-10-14T01:28:38.912127+0000","mapping_epoch":13,"log_start":"157'324009","ondisk_log_start":"157'324009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3719438605,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":328726,"num_read_kb":1277992,"num_write":323893,"num_write_kb":1295358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"159'339707","reported_seq":676284,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:38.884008+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:28:38.884008+0000","last_peered":"2023-10-14T01:28:38.884008+0000","last_clean":"2023-10-14T01:28:38.884008+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:28:38.884008+0000","last_undegraded":"2023-10-14T01:28:38.884008+0000","last_fullsized":"2023-10-14T01:28:38.884008+0000","mapping_epoch":13,"log_start":"157'329611","ondisk_log_start":"157'329611","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3772653817,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":336907,"num_read_kb":1304492,"num_write":327408,"num_write_kb":1309358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"159'332578","reported_seq":657588,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:37.987713+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:28:37.987713+0000","last_peered":"2023-10-14T01:28:37.987713+0000","last_clean":"2023-10-14T01:28:37.987713+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:28:37.987713+0000","last_undegraded":"2023-10-14T01:28:37.987713+0000","last_fullsized":"2023-10-14T01:28:37.987713+0000","mapping_epoch":13,"log_start":"157'322509","ondisk_log_start":"157'322509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3702755512,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":325234,"num_read_kb":1274427,"num_write":321399,"num_write_kb":1285483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":375,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:27:44.961218+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:27:44.961218+0000","last_peered":"2023-10-14T01:27:44.961218+0000","last_clean":"2023-10-14T01:27:44.961218+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:27:44.961218+0000","last_undegraded":"2023-10-14T01:27:44.961218+0000","last_fullsized":"2023-10-14T01:27:44.961218+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29829171512,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2629571,"num_read_kb":10265834,"num_write":2592490,"num_write_kb":10368489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21235093504,"data_stored":25546551920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80439,"ondisk_log_size":80439,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739477,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4703036,"kb_used_data":3891124,"kb_used_omap":336,"kb_used_meta":811567,"kb_avail":89668804,"statfs":{"total":96636764160,"available":91820855296,"internally_reserved":0,"allocated":3984510976,"data_stored":4793595527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":831045591},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.79100000000000004,"15min":0.82499999999999996},"min":{"1min":0.189,"5min":0.189,"15min":0.17199999999999999},"max":{"1min":4.0279999999999996,"5min":12.356,"15min":12.356},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.52000000000000002,"15min":0.62},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.191},"max":{"1min":1.343,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.252}]},{"osd":1,"last update":"Sat Oct 14 01:27:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.2649999999999999,"15min":1.282},"min":{"1min":0.248,"5min":0.20200000000000001,"15min":0.17999999999999999},"max":{"1min":3.8929999999999998,"5min":11.956,"15min":14.005000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.333,"15min":1.4079999999999999},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.7639999999999998,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.311}]}]},{"osd":1,"up_from":8,"seq":34359739459,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12221728,"kb_used_data":10368984,"kb_used_omap":864,"kb_used_meta":1851871,"kb_avail":82150112,"statfs":{"total":96636764160,"available":84121714688,"internally_reserved":0,"allocated":10617839616,"data_stored":12773224225,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":1896316410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,8,7,3,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":1.0369999999999999,"15min":0.90800000000000003},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.183},"max":{"1min":3.9710000000000001,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.752,"5min":1.175,"15min":1.1180000000000001},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.20899999999999999},"max":{"1min":5.3220000000000001,"5min":15.147,"15min":24.908000000000001},"last":0.51600000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:28:10 2023","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.298,"15min":1.3109999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23499999999999999},"max":{"1min":12.818,"5min":12.818,"15min":27.837},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.327,"15min":1.353},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":9.7669999999999995,"5min":15.09,"15min":25.710000000000001},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739474,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8360980,"kb_used_data":6480204,"kb_used_omap":495,"kb_used_meta":1880272,"kb_avail":86010860,"statfs":{"total":96636764160,"available":88075120640,"internally_reserved":0,"allocated":6635728896,"data_stored":7981136995,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":1925398697},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:49 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.76500000000000001,"15min":0.93999999999999995},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.1950000000000003,"5min":5.3250000000000002,"15min":11.734999999999999},"last":0.248},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.67200000000000004,"15min":0.63900000000000001},"min":{"1min":0.221,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":2.0379999999999998,"5min":5.7619999999999996,"15min":6.5110000000000001},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:27:49 2023","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.82799999999999996,"15min":0.876},"min":{"1min":0.23300000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":4.798,"5min":5.4610000000000003,"15min":9.8059999999999992},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66400000000000003,"15min":0.63100000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":4.024,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":0.21199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6634885120,"data_stored":7980639866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10616487936,"data_stored":12772227384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3983720448,"data_stored":4793684670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:28:42.129 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:42.129 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:42.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:42.415+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:42.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:42.447+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:42.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:42.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:42.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:42.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:42.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:42.479+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:42.875 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.871+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:28:42.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:28:42.893 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.891+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:42.900 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.899+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:42.914 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:42.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:42.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:42.915 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:42.911+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:43.394 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:43.390+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:43.428 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:43.418+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:43.498 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:43.490+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:43.501 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:43.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:43.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:43.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:43.502 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:43.498+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:43.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:43.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:43.555 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:43.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:43.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:43.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.566+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:43.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:43.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:43.914 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:43.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:43.915 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.910+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:43.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:43.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:43.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:44.692 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:44.690+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:28:44.714 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:44.706+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:44.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:44.714+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:44.716 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:44.714+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:44.717 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:44.714+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:44.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:44.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:44.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:44.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:44.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:44.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:44.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:44.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:44.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:44.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:44.826 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:44.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:47.250 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:47.246+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:47.262 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:47.258+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:47.273 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:47.266+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:47.283 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:47.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:47.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:47.282+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:48.008 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:48.042 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:48.043 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.038+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:48.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:28:48.273 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.270+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:28:48.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:48.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:48.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:48.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:48.279 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:48.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:48.308 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.306+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:48.321 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.318+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:48.449 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:48.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.454+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:48.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:48.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:48.481 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.474+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:48.516 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.514+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:48.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:48.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:48.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:48.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:48.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:48.581 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:48.841 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.838+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:48.860 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:48.858+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:49.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.206+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:49.220 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:49.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:49.221 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:49.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:49.222 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.218+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:49.465 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:28:49.466 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.462+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:28:49.482 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:49.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:49.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:49.478+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:49.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.482+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:28:49.506 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:49.507 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:49.502+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:49.542 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:49.538+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:50.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:50.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:50.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:50.470+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:50.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:50.490+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:50.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:50.498+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:50.511 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:50.506+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:54.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:54.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:54.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:54.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:54.538 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:54.534+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:54.552 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:54.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:54.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:54.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:54.553 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:54.550+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:55.880 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:55.874+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:55.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:55.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:55.886 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:55.878+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:57.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:57.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:57.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:57.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:57.710 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:57.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:57.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:57.702+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:57.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:57.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:57.711 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:57.706+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:57.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:57.730+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:58.032 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:58.030+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:28:58.074 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:58.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:28:58.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:58.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:28:58.075 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:28:58.070+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:28:58.754 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:58.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:58.755 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:58.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:58.780 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:58.774+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:28:58.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:58.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:28:58.832 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:58.830+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:28:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:59.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:28:59.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:59.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:28:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:59.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:28:59.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:59.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:28:59.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:59.686+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:28:59.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:28:59.698+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:00.653 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.642+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:00.655 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.654+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:00.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:00.674 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:00.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:00.716 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.714+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:00.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.722+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:00.983 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.978+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:00.992 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.990+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:01.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:00.998+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:01.003 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:01.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:01.004 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:01.002+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:01.007 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:01.006+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:01.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:01.018+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:01.076 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:01.074+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:02.130 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:02.688 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:02.686+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:03.059 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:29:03.060 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:29:03.181 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2944,"stamp":"2023-10-14T01:29:01.990358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29865175896,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2638544,"num_read_kb":10301251,"num_write":2601547,"num_write_kb":10406589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80446,"ondisk_log_size":80446,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25681824,"kb_used_data":20809160,"kb_used_omap":1696,"kb_used_meta":4870943,"kb_avail":257433696,"statfs":{"total":289910292480,"available":263612104704,"internally_reserved":0,"allocated":21308579840,"data_stored":25617976781,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1737094,"internal_metadata":4987846266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,7,6,17,14,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18010112,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4646,"num_read_kb":18556,"num_write":4536,"num_write_kb":18144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005127"},"pg_stats":[{"pgid":"2.7","version":"160'349322","reported_seq":685106,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:58.476486+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:28:58.476486+0000","last_peered":"2023-10-14T01:28:58.476486+0000","last_clean":"2023-10-14T01:28:58.476486+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:28:58.476486+0000","last_undegraded":"2023-10-14T01:28:58.476486+0000","last_fullsized":"2023-10-14T01:28:58.476486+0000","mapping_epoch":13,"log_start":"157'339304","ondisk_log_start":"157'339304","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1110,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 564s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3776655798,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":336152,"num_read_kb":1306799,"num_write":328444,"num_write_kb":1313534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'349682","reported_seq":685924,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.476911+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:28:58.476911+0000","last_peered":"2023-10-14T01:28:58.476911+0000","last_clean":"2023-10-14T01:28:58.476911+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:28:58.476911+0000","last_undegraded":"2023-10-14T01:28:58.476911+0000","last_fullsized":"2023-10-14T01:28:58.476911+0000","mapping_epoch":13,"log_start":"157'339613","ondisk_log_start":"157'339613","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3770929520,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":338154,"num_read_kb":1304829,"num_write":328767,"num_write_kb":1314695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'316394","reported_seq":629029,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.460464+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:28:58.460464+0000","last_peered":"2023-10-14T01:28:58.460464+0000","last_clean":"2023-10-14T01:28:58.460464+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:28:58.460464+0000","last_undegraded":"2023-10-14T01:28:58.460464+0000","last_fullsized":"2023-10-14T01:28:58.460464+0000","mapping_epoch":13,"log_start":"157'306308","ondisk_log_start":"157'306308","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3608162706,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":312813,"num_read_kb":1234984,"num_write":315055,"num_write_kb":1260190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'308616","reported_seq":617018,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.439147+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:28:58.439147+0000","last_peered":"2023-10-14T01:28:58.439147+0000","last_clean":"2023-10-14T01:28:58.439147+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:28:58.439147+0000","last_undegraded":"2023-10-14T01:28:58.439147+0000","last_fullsized":"2023-10-14T01:28:58.439147+0000","mapping_epoch":13,"log_start":"157'298607","ondisk_log_start":"157'298607","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3529310339,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":308638,"num_read_kb":1216621,"num_write":307340,"num_write_kb":1229282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"160'348413","reported_seq":696817,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:28:59.571958+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:28:59.571958+0000","last_peered":"2023-10-14T01:28:59.571958+0000","last_clean":"2023-10-14T01:28:59.571958+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:28:59.571958+0000","last_undegraded":"2023-10-14T01:28:59.571958+0000","last_fullsized":"2023-10-14T01:28:59.571958+0000","mapping_epoch":13,"log_start":"157'338409","ondisk_log_start":"157'338409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3971817791,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":348644,"num_read_kb":1368442,"num_write":345836,"num_write_kb":1383197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"160'335152","reported_seq":664654,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.990481+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:28:58.990481+0000","last_peered":"2023-10-14T01:28:58.990481+0000","last_clean":"2023-10-14T01:28:58.990481+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:28:58.990481+0000","last_undegraded":"2023-10-14T01:28:58.990481+0000","last_fullsized":"2023-10-14T01:28:58.990481+0000","mapping_epoch":13,"log_start":"157'325109","ondisk_log_start":"157'325109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3723776269,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":329803,"num_read_kb":1282296,"num_write":324982,"num_write_kb":1299714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"160'340875","reported_seq":678608,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.963702+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:28:58.963702+0000","last_peered":"2023-10-14T01:28:58.963702+0000","last_clean":"2023-10-14T01:28:58.963702+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:28:58.963702+0000","last_undegraded":"2023-10-14T01:28:58.963702+0000","last_fullsized":"2023-10-14T01:28:58.963702+0000","mapping_epoch":13,"log_start":"157'330811","ondisk_log_start":"157'330811","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3777155321,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":338035,"num_read_kb":1308984,"num_write":328536,"num_write_kb":1313870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"160'333635","reported_seq":659600,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.320487+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:28:58.320487+0000","last_peered":"2023-10-14T01:28:58.320487+0000","last_clean":"2023-10-14T01:28:58.320487+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:28:58.320487+0000","last_undegraded":"2023-10-14T01:28:58.320487+0000","last_fullsized":"2023-10-14T01:28:58.320487+0000","mapping_epoch":13,"log_start":"157'323609","ondisk_log_start":"157'323609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3706777784,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":326179,"num_read_kb":1278187,"num_write":322418,"num_write_kb":1289559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":376,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.489789+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:28:17.489789+0000","last_peered":"2023-10-14T01:28:17.489789+0000","last_clean":"2023-10-14T01:28:17.489789+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:28:17.489789+0000","last_undegraded":"2023-10-14T01:28:17.489789+0000","last_fullsized":"2023-10-14T01:28:17.489789+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29864585528,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2638418,"num_read_kb":10301142,"num_write":2601378,"num_write_kb":10404041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21306650624,"data_stored":25617646192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80319,"ondisk_log_size":80319,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739481,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4751912,"kb_used_data":3903904,"kb_used_omap":336,"kb_used_meta":847663,"kb_avail":89619928,"statfs":{"total":96636764160,"available":91770806272,"internally_reserved":0,"allocated":3997597696,"data_stored":4806575389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":868007895},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83999999999999997,"15min":0.80900000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.17199999999999999},"max":{"1min":4.8280000000000003,"5min":12.356,"15min":12.356},"last":0.253},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.47599999999999998,"15min":0.61699999999999999},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.57499999999999996,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.70299999999999996}]},{"osd":1,"last update":"Sat Oct 14 01:28:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.1080000000000001,"15min":1.262},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.17999999999999999},"max":{"1min":3.1560000000000001,"5min":11.956,"15min":14.005000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":1.2889999999999999,"15min":1.3480000000000001},"min":{"1min":0.26000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":3.2000000000000002,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.26700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739463,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12470892,"kb_used_data":10403364,"kb_used_omap":864,"kb_used_meta":2066655,"kb_avail":81900948,"statfs":{"total":96636764160,"available":83866570752,"internally_reserved":0,"allocated":10653044736,"data_stored":12808191415,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":2116255226},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,7,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.0229999999999999,"15min":0.95699999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.863,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.377},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.212,"15min":1.2230000000000001},"min":{"1min":0.24099999999999999,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":12.893000000000001,"5min":15.147,"15min":24.908000000000001},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:28:10 2023","interfaces":[{"interface":"back","average":{"1min":1.7450000000000001,"5min":1.298,"15min":1.3109999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23499999999999999},"max":{"1min":12.818,"5min":12.818,"15min":27.837},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.327,"15min":1.353},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":9.7669999999999995,"5min":15.09,"15min":25.710000000000001},"last":1.478}]}]},{"osd":0,"up_from":8,"seq":34359739478,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8459020,"kb_used_data":6501892,"kb_used_omap":495,"kb_used_meta":1956624,"kb_avail":85912820,"statfs":{"total":96636764160,"available":87974727680,"internally_reserved":0,"allocated":6657937408,"data_stored":8003209977,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":2003583145},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,3,7,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.74199999999999999,"15min":0.90200000000000002},"min":{"1min":0.215,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.4420000000000002,"5min":5.3250000000000002,"15min":11.734999999999999},"last":3.8660000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59499999999999997,"15min":0.62},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":2.169,"5min":5.306,"15min":6.5110000000000001},"last":0.25800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:28:52 2023","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.0669999999999999,"15min":0.95699999999999996},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":9.8059999999999992},"last":3.7269999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73299999999999998,"15min":0.621},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":3.9359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6657495040,"data_stored":8003122810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10652512256,"data_stored":12808014136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3996643328,"data_stored":4806509246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:29:03.184 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:03.185 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:04.030 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:04.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.014+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:04.031 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.022+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:04.036 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.034+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:04.051 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.046+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:04.241 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:04.242 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.238+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:04.248 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:04.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:04.249 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.246+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:04.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:04.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:04.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:04.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:05.259 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:05.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:05.318 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:05.314+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:05.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:05.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:05.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:05.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:05.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:05.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:05.582 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:05.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:05.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:05.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:05.583 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:05.578+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:05.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:05.762+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:05.793 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:05.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:07.334 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.330+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:07.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:07.347 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:07.348 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.342+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:07.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:07.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:07.349 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:07.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:07.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.790+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:07.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:07.822 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:07.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:07.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.818+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:07.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:07.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:10.556 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:10.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:10.588 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:10.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:10.589 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:10.586+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:11.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:11.742+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:11.759 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:11.750+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:11.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:11.766+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:12.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:12.622+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:12.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:12.626+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:13.812 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:13.810+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:13.816 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:13.814+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:13.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:13.850+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:14.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:14.110 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.102+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:14.128 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:14.301 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:14.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:14.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:14.307 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.302+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:14.325 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.322+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:14.350 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:14.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:14.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:14.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.926+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:14.935 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:14.939 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:14.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:14.938+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:15.763 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:15.758+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:15.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:15.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:15.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:15.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:15.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:15.782+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:15.827 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:15.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:19.330 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:19.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:19.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:19.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:19.337 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:19.326+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:19.373 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:19.370+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:19.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:19.386+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:19.397 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:19.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:20.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:20.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:20.727 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:20.726+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:20.734 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:20.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:20.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:20.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:20.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:20.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:20.735 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:20.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:20.736 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:20.730+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:21.096 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:21.078+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:21.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:21.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:21.129 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:21.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:21.130 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:21.126+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:21.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:21.134+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:21.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:21.158+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:21.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:21.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:21.315 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:21.310+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:21.317 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:21.314+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:21.321 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:21.318+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:21.327 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:21.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:21.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:21.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:21.328 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:21.326+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:22.959 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:22.958+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:22.987 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:22.986+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:23.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:23.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:23.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:23.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:23.046 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:23.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:23.047 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:23.042+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:23.186 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:24.283 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:29:24.284 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:29:24.328 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2955,"stamp":"2023-10-14T01:29:23.997633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29900102488,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2647337,"num_read_kb":10336391,"num_write":2610638,"num_write_kb":10442953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80570,"ondisk_log_size":80570,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":26021192,"kb_used_data":20878444,"kb_used_omap":1696,"kb_used_meta":5141023,"kb_avail":257094328,"statfs":{"total":289910292480,"available":263264591872,"internally_reserved":0,"allocated":21379526656,"data_stored":25687444941,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1737094,"internal_metadata":5264408186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,7,4,12,19,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16982016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4560,"num_read_kb":18228,"num_write":4527,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005634"},"pg_stats":[{"pgid":"2.7","version":"160'350557","reported_seq":687471,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:29:19.158595+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:29:19.158595+0000","last_peered":"2023-10-14T01:29:19.158595+0000","last_clean":"2023-10-14T01:29:19.158595+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:29:19.158595+0000","last_undegraded":"2023-10-14T01:29:19.158595+0000","last_fullsized":"2023-10-14T01:29:19.158595+0000","mapping_epoch":13,"log_start":"157'340504","ondisk_log_start":"157'340504","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1142,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 584s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3781108150,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":337261,"num_read_kb":1311235,"num_write":329585,"num_write_kb":1318098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'350878","reported_seq":688188,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.125995+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:29:19.125995+0000","last_peered":"2023-10-14T01:29:19.125995+0000","last_clean":"2023-10-14T01:29:19.125995+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:29:19.125995+0000","last_undegraded":"2023-10-14T01:29:19.125995+0000","last_fullsized":"2023-10-14T01:29:19.125995+0000","mapping_epoch":13,"log_start":"157'340813","ondisk_log_start":"157'340813","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3775271280,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":339206,"num_read_kb":1309033,"num_write":329891,"num_write_kb":1319191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'317506","reported_seq":631283,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.099945+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:29:19.099945+0000","last_peered":"2023-10-14T01:29:19.099945+0000","last_clean":"2023-10-14T01:29:19.099945+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:29:19.099945+0000","last_undegraded":"2023-10-14T01:29:19.099945+0000","last_fullsized":"2023-10-14T01:29:19.099945+0000","mapping_epoch":13,"log_start":"157'307408","ondisk_log_start":"157'307408","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3612389778,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":313951,"num_read_kb":1239528,"num_write":316167,"num_write_kb":1264638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'309783","reported_seq":619243,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.115293+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:29:19.115293+0000","last_peered":"2023-10-14T01:29:19.115293+0000","last_clean":"2023-10-14T01:29:19.115293+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:29:19.115293+0000","last_undegraded":"2023-10-14T01:29:19.115293+0000","last_fullsized":"2023-10-14T01:29:19.115293+0000","mapping_epoch":13,"log_start":"157'299707","ondisk_log_start":"157'299707","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3533709443,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":309695,"num_read_kb":1220845,"num_write":308507,"num_write_kb":1233950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"160'349523","reported_seq":699003,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.719396+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:29:19.719396+0000","last_peered":"2023-10-14T01:29:19.719396+0000","last_clean":"2023-10-14T01:29:19.719396+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:29:19.719396+0000","last_undegraded":"2023-10-14T01:29:19.719396+0000","last_fullsized":"2023-10-14T01:29:19.719396+0000","mapping_epoch":13,"log_start":"157'339509","ondisk_log_start":"157'339509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3976102207,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":349720,"num_read_kb":1372746,"num_write":346943,"num_write_kb":1387625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"160'336263","reported_seq":666847,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.179933+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:29:19.179933+0000","last_peered":"2023-10-14T01:29:19.179933+0000","last_clean":"2023-10-14T01:29:19.179933+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:29:19.179933+0000","last_undegraded":"2023-10-14T01:29:19.179933+0000","last_fullsized":"2023-10-14T01:29:19.179933+0000","mapping_epoch":13,"log_start":"157'326209","ondisk_log_start":"157'326209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3728023821,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":330872,"num_read_kb":1286564,"num_write":326081,"num_write_kb":1304110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"160'342085","reported_seq":681016,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.135283+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:29:19.135283+0000","last_peered":"2023-10-14T01:29:19.135283+0000","last_clean":"2023-10-14T01:29:19.135283+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:29:19.135283+0000","last_undegraded":"2023-10-14T01:29:19.135283+0000","last_fullsized":"2023-10-14T01:29:19.135283+0000","mapping_epoch":13,"log_start":"157'332011","ondisk_log_start":"157'332011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3781767417,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":339213,"num_read_kb":1313696,"num_write":329731,"num_write_kb":1318650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"160'334818","reported_seq":661905,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.120638+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:29:19.120638+0000","last_peered":"2023-10-14T01:29:19.120638+0000","last_clean":"2023-10-14T01:29:19.120638+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:29:19.120638+0000","last_undegraded":"2023-10-14T01:29:19.120638+0000","last_fullsized":"2023-10-14T01:29:19.120638+0000","mapping_epoch":13,"log_start":"157'324809","ondisk_log_start":"157'324809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3711140024,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":327293,"num_read_kb":1282635,"num_write":323564,"num_write_kb":1294143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":376,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.489789+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:28:17.489789+0000","last_peered":"2023-10-14T01:28:17.489789+0000","last_clean":"2023-10-14T01:28:17.489789+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:28:17.489789+0000","last_undegraded":"2023-10-14T01:28:17.489789+0000","last_fullsized":"2023-10-14T01:28:17.489789+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29899512120,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2647211,"num_read_kb":10336282,"num_write":2610469,"num_write_kb":10440405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21376192512,"data_stored":25685676656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80443,"ondisk_log_size":80443,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739485,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4833256,"kb_used_data":3917280,"kb_used_omap":336,"kb_used_meta":915631,"kb_avail":89538584,"statfs":{"total":96636764160,"available":91687510016,"internally_reserved":0,"allocated":4011294720,"data_stored":4819993885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":937607127},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,3,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83999999999999997,"15min":0.80900000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.17199999999999999},"max":{"1min":4.8280000000000003,"5min":12.356,"15min":12.356},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.47599999999999998,"15min":0.61699999999999999},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.57499999999999996,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":2.6739999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:28:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.1080000000000001,"15min":1.262},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.17999999999999999},"max":{"1min":3.1560000000000001,"5min":11.956,"15min":14.005000000000001},"last":3.145},{"interface":"front","average":{"1min":0.78100000000000003,"5min":1.2889999999999999,"15min":1.3480000000000001},"min":{"1min":0.26000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":3.2000000000000002,"5min":13.342000000000001,"15min":13.342000000000001},"last":4.2169999999999996}]}]},{"osd":1,"up_from":8,"seq":34359739467,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12623316,"kb_used_data":10437768,"kb_used_omap":864,"kb_used_meta":2184671,"kb_avail":81748524,"statfs":{"total":96636764160,"available":83710488576,"internally_reserved":0,"allocated":10688274432,"data_stored":12842704311,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":885254,"internal_metadata":2237103610},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,1,6,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.0229999999999999,"15min":0.95699999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.863,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.252},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.212,"15min":1.2230000000000001},"min":{"1min":0.24099999999999999,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":12.893000000000001,"5min":15.147,"15min":24.908000000000001},"last":1.369}]},{"osd":2,"last update":"Sat Oct 14 01:29:11 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.1819999999999999,"15min":1.3069999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.8929999999999998,"5min":12.818,"15min":27.837},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.179,"15min":1.304},"min":{"1min":0.219,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":4.4630000000000001,"5min":15.09,"15min":25.710000000000001},"last":0.377}]}]},{"osd":0,"up_from":8,"seq":34359739482,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8564620,"kb_used_data":6523396,"kb_used_omap":495,"kb_used_meta":2040720,"kb_avail":85807220,"statfs":{"total":96636764160,"available":87866593280,"internally_reserved":0,"allocated":6679957504,"data_stored":8024746745,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":507735,"internal_metadata":2089697449},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.74199999999999999,"15min":0.90200000000000002},"min":{"1min":0.215,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.4420000000000002,"5min":5.3250000000000002,"15min":11.734999999999999},"last":0.438},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59499999999999997,"15min":0.62},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":2.169,"5min":5.306,"15min":6.5110000000000001},"last":0.29299999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:28:52 2023","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.0669999999999999,"15min":0.95699999999999996},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":9.8059999999999992},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73299999999999998,"15min":0.621},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6678581248,"data_stored":8023725690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10687815680,"data_stored":12842563896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4009795584,"data_stored":4819387070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:29:24.329 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:24.330 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:24.799 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:24.794+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:24.805 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:24.798+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:24.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:24.818+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:24.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:24.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:24.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:24.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:24.839 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:24.834+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:25.354 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.346+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:25.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:25.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.358+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:25.367 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.362+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:25.368 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.366+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:25.398 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.394+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:25.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:25.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:25.678 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:25.674+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:26.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:26.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:26.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:26.374+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:26.394 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:26.390+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:26.690 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:26.682+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:26.699 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:26.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:26.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:26.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:26.700 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:26.694+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:27.490 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:27.486+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:27.491 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:27.486+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:27.547 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:27.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:27.548 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:27.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:27.548 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:27.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:28.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:28.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:28.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:28.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:28.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.554+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:28.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.570+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:28.579 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.578+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:28.611 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:28.610+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:29.221 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.218+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:29.231 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:29.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:29.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.446+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:29.463 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.458+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:29.815 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.806+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:29.823 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:29.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:29.824 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.822+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:29.849 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:29.846+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:30.100 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:30.094+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:31.610 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.606+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:29:31.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.670+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:31.731 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:31.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:31.732 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.726+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:31.744 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.734+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:31.748 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:31.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:31.746+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:32.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:32.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:32.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.206+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:32.261 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.258+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:32.263 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:32.270 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.262+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:32.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:32.773 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:32.774 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.770+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:32.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:32.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.786+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:32.807 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.802+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:32.889 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.882+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:32.938 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:32.934+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:35.543 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:35.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:35.544 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:35.542+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:35.551 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:35.550+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:35.562 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:35.558+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:35.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:35.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:35.563 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:35.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:35.564 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:35.562+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:37.172 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:37.170+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:37.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:37.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:37.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:37.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:37.233 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:37.230+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:37.278 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:37.274+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:37.298 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:37.294+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:37.302 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:37.298+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:39.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:39.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.169+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:39.189 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:39.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:39.190 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.185+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:39.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:39.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:39.198 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:39.339 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.337+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:39.346 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.341+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:39.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:39.379 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:39.380 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:39.396 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:39.405 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.401+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:39.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:39.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:40.382 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:40.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.393+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:40.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:40.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:40.436 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.433+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:40.447 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:40.464 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.461+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:40.476 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:40.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:29:40.498 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.489+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:40.506 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:40.514 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:40.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:40.515 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.505+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:40.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:40.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:40.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:40.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:41.191 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:41.192 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:41.204 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.201+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:41.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:41.213 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:41.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:41.703 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.701+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:41.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:41.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.725+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:41.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:41.776 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:41.792 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:41.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:42.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:42.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:42.015 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.009+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:42.032 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.025+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:42.048 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.045+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:42.053 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.049+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:42.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.137+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:42.156 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:42.176 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:42.178 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.173+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:42.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:42.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:42.249 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:42.245+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:42.268 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:42.265+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:42.269 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:42.265+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:42.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:42.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:42.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:42.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:42.287 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:42.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:42.288 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:42.281+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:43.038 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:43.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:43.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:43.039 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:43.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:43.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:43.040 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.033+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:43.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:29:43.952 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:43.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:43.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:43.953 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:43.949+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:44.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:44.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:44.223 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:44.217+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:44.237 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:44.233+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:44.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:44.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:44.247 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:44.241+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:44.331 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:44.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:44.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:44.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:44.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:44.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:44.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:44.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:44.577+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:44.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:44.585+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:44.629 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:44.625+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:29:45.148 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:45.149 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.145+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:45.157 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.153+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:45.177 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.173+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:45.192 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:29:45.192 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:29:45.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.189+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:45.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.189+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:45.195 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.189+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:45.223 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2965,"stamp":"2023-10-14T01:29:44.006804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29929331544,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2655366,"num_read_kb":10368471,"num_write":2619026,"num_write_kb":10476505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80640,"ondisk_log_size":80640,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25449016,"kb_used_data":20934812,"kb_used_omap":1747,"kb_used_meta":4512428,"kb_avail":257666504,"statfs":{"total":289910292480,"available":263850500096,"internally_reserved":0,"allocated":21437247488,"data_stored":25742028237,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1789654,"internal_metadata":4620726570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,3,10,8,18,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14086144,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3997,"num_read_kb":15964,"num_write":4115,"num_write_kb":16460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007817"},"pg_stats":[{"pgid":"2.7","version":"160'351699","reported_seq":689642,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:29:39.786459+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:29:39.786459+0000","last_peered":"2023-10-14T01:29:39.786459+0000","last_clean":"2023-10-14T01:29:39.786459+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:29:39.786459+0000","last_undegraded":"2023-10-14T01:29:39.786459+0000","last_fullsized":"2023-10-14T01:29:39.786459+0000","mapping_epoch":13,"log_start":"158'341604","ondisk_log_start":"158'341604","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1192,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 605s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3784712630,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":338263,"num_read_kb":1315235,"num_write":330618,"num_write_kb":1322230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'351946","reported_seq":690226,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.745033+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:29:39.745033+0000","last_peered":"2023-10-14T01:29:39.745033+0000","last_clean":"2023-10-14T01:29:39.745033+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:29:39.745033+0000","last_undegraded":"2023-10-14T01:29:39.745033+0000","last_fullsized":"2023-10-14T01:29:39.745033+0000","mapping_epoch":13,"log_start":"158'341913","ondisk_log_start":"158'341913","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3778720112,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":340157,"num_read_kb":1312837,"num_write":330887,"num_write_kb":1323175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'318600","reported_seq":633511,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.752303+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:29:39.752303+0000","last_peered":"2023-10-14T01:29:39.752303+0000","last_clean":"2023-10-14T01:29:39.752303+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:29:39.752303+0000","last_undegraded":"2023-10-14T01:29:39.752303+0000","last_fullsized":"2023-10-14T01:29:39.752303+0000","mapping_epoch":13,"log_start":"157'308508","ondisk_log_start":"157'308508","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3616166290,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":315084,"num_read_kb":1244048,"num_write":317261,"num_write_kb":1269014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'310911","reported_seq":621415,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.758150+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:29:39.758150+0000","last_peered":"2023-10-14T01:29:39.758150+0000","last_clean":"2023-10-14T01:29:39.758150+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:29:39.758150+0000","last_undegraded":"2023-10-14T01:29:39.758150+0000","last_fullsized":"2023-10-14T01:29:39.758150+0000","mapping_epoch":13,"log_start":"157'300907","ondisk_log_start":"157'300907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3537559683,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":310738,"num_read_kb":1225017,"num_write":309635,"num_write_kb":1238462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"160'350493","reported_seq":700916,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.863145+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:29:39.863145+0000","last_peered":"2023-10-14T01:29:39.863145+0000","last_clean":"2023-10-14T01:29:39.863145+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:29:39.863145+0000","last_undegraded":"2023-10-14T01:29:39.863145+0000","last_fullsized":"2023-10-14T01:29:39.863145+0000","mapping_epoch":13,"log_start":"158'340409","ondisk_log_start":"158'340409","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3979379007,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":350654,"num_read_kb":1376482,"num_write":347911,"num_write_kb":1391497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"160'337284","reported_seq":668800,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.202404+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:29:39.202404+0000","last_peered":"2023-10-14T01:29:39.202404+0000","last_clean":"2023-10-14T01:29:39.202404+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:29:39.202404+0000","last_undegraded":"2023-10-14T01:29:39.202404+0000","last_fullsized":"2023-10-14T01:29:39.202404+0000","mapping_epoch":13,"log_start":"158'327209","ondisk_log_start":"158'327209","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3731501325,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":331788,"num_read_kb":1290216,"num_write":327074,"num_write_kb":1308082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"160'343050","reported_seq":682883,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.229179+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:29:39.229179+0000","last_peered":"2023-10-14T01:29:39.229179+0000","last_clean":"2023-10-14T01:29:39.229179+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:29:39.229179+0000","last_undegraded":"2023-10-14T01:29:39.229179+0000","last_fullsized":"2023-10-14T01:29:39.229179+0000","mapping_epoch":13,"log_start":"158'333011","ondisk_log_start":"158'333011","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3785027833,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":340095,"num_read_kb":1317224,"num_write":330661,"num_write_kb":1322370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"160'336100","reported_seq":664363,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.716257+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:29:39.716257+0000","last_peered":"2023-10-14T01:29:39.716257+0000","last_clean":"2023-10-14T01:29:39.716257+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:29:39.716257+0000","last_undegraded":"2023-10-14T01:29:39.716257+0000","last_fullsized":"2023-10-14T01:29:39.716257+0000","mapping_epoch":13,"log_start":"158'326009","ondisk_log_start":"158'326009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3715674296,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":328461,"num_read_kb":1287303,"num_write":324810,"num_write_kb":1299127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":376,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.489789+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:28:17.489789+0000","last_peered":"2023-10-14T01:28:17.489789+0000","last_clean":"2023-10-14T01:28:17.489789+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:28:17.489789+0000","last_undegraded":"2023-10-14T01:28:17.489789+0000","last_fullsized":"2023-10-14T01:28:17.489789+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29928741176,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2655240,"num_read_kb":10368362,"num_write":2618857,"num_write_kb":10473957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21434634240,"data_stored":25740857968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80513,"ondisk_log_size":80513,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739489,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4895304,"kb_used_data":3927744,"kb_used_omap":336,"kb_used_meta":967215,"kb_avail":89476536,"statfs":{"total":96636764160,"available":91623972864,"internally_reserved":0,"allocated":4022009856,"data_stored":4830070045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":990429143},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.83999999999999997,"15min":0.80900000000000005},"min":{"1min":0.187,"5min":0.187,"15min":0.17199999999999999},"max":{"1min":4.8280000000000003,"5min":12.356,"15min":12.356},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.47599999999999998,"15min":0.61699999999999999},"min":{"1min":0.20899999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.57499999999999996,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.30399999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:28:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.1080000000000001,"15min":1.262},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.17999999999999999},"max":{"1min":3.1560000000000001,"5min":11.956,"15min":14.005000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":1.2889999999999999,"15min":1.3480000000000001},"min":{"1min":0.26000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":3.2000000000000002,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739471,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11924964,"kb_used_data":10466460,"kb_used_omap":866,"kb_used_meta":1457629,"kb_avail":82446876,"statfs":{"total":96636764160,"available":84425601024,"internally_reserved":0,"allocated":10717655040,"data_stored":12870499767,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":886933,"internal_metadata":1492612971},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,9,3,9,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.0229999999999999,"15min":0.95699999999999996},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.863,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.212,"15min":1.2230000000000001},"min":{"1min":0.24099999999999999,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":12.893000000000001,"5min":15.147,"15min":24.908000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:29:11 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.1819999999999999,"15min":1.3069999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.8929999999999998,"5min":12.818,"15min":27.837},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.179,"15min":1.304},"min":{"1min":0.219,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":4.4630000000000001,"5min":15.09,"15min":25.710000000000001},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739486,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8628748,"kb_used_data":6540608,"kb_used_omap":545,"kb_used_meta":2087582,"kb_avail":85743092,"statfs":{"total":96636764160,"available":87800926208,"internally_reserved":0,"allocated":6697582592,"data_stored":8041458425,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":558616,"internal_metadata":2137684456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.74199999999999999,"15min":0.90200000000000002},"min":{"1min":0.215,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.4420000000000002,"5min":5.3250000000000002,"15min":11.734999999999999},"last":3.0230000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.59499999999999997,"15min":0.62},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.17899999999999999},"max":{"1min":2.169,"5min":5.306,"15min":6.5110000000000001},"last":0.311}]},{"osd":2,"last update":"Sat Oct 14 01:28:52 2023","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.0669999999999999,"15min":0.95699999999999996},"min":{"1min":0.22,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":8.8049999999999997,"5min":8.8049999999999997,"15min":9.8059999999999992},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.73299999999999998,"15min":0.621},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.7729999999999997,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6696267776,"data_stored":8040482426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10717396992,"data_stored":12870498616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4020969472,"data_stored":4829876926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:29:45.224 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:45.225 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-14T01:29:45.861 INFO:teuthology.orchestra.run.smithi138.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-14T01:29:45.915 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:45.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:45.981 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:45.977+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:46.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:46.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:46.017 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:46.009+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:46.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:46.017+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:46.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:46.017+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:46.020 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:46.017+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:47.322 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:47.317+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:47.336 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:47.329+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:47.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:47.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:29:47.973 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:47.969+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:29:47.988 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:47.981+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:47.991 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:47.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:29:47.995 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:47.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:47.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:47.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:47.996 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:47.989+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:49.089 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.085+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:49.112 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.109+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:49.138 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:49.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:49.139 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.117+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:49.154 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.149+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:49.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:49.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:49.182 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.177+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:49.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.193+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:49.671 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.665+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:49.688 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:49.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:49.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:49.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:49.689 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:49.831 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:49.842 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:49.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:49.843 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.837+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:49.857 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:49.853+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:50.458 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:50.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:50.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:50.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:50.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:50.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:50.459 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:50.445+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:50.471 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:50.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:51.376 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:51.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:51.377 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.373+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:51.408 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.405+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:51.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:51.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.517+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:51.529 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:51.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:51.530 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:51.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.557+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:51.817 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.813+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:51.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:51.839 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:51.840 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.833+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:51.852 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:51.845+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:55.765 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:55.761+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:29:55.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:55.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:29:55.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:55.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:29:55.786 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:29:55.781+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:29:57.124 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.121+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:29:57.158 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:29:57.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:29:57.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:29:57.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:29:57.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:29:57.166 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:29:57.167 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:29:57.161+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:00.333 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:00.329+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:00.390 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:00.385+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:00.391 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:00.385+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:00.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:00.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:00.398 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:00.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:30:00.399 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:00.393+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:01.691 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:01.689+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:01.692 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:01.693+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:01.698 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:01.697+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:01.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:01.957 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:01.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:01.974 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:01.973+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:02.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:02.481+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:02.485 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:02.485+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:02.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:02.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:02.496 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:02.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:02.497 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:02.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:02.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:02.501+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:03.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:03.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:03.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:03.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:03.643 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:03.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:03.672 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:03.669+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:04.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:04.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:04.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:04.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:04.521 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:04.521+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:04.574 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:04.573+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:04.662 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:04.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:30:04.663 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:04.661+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:04.665 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:04.665+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:04.668 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:04.669+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:04.677 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:04.677+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:04.682 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:04.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:30:04.683 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:04.681+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:04.947 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:04.949+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:04.963 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:04.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:05.505 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:05.505+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:05.510 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:05.509+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:05.519 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:05.521+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:05.524 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:05.525+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:05.525 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:05.525+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:05.608 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:05.630 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:05.642 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.641+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:05.644 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:05.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.645+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:05.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.653+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:05.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.737+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:05.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:05.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:05.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:05.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:05.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:05.767 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:05.768 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:05.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:05.917 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:30:06.557 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:30:06.558 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:30:06.586 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2976,"stamp":"2023-10-14T01:30:06.011348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29963205464,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2665352,"num_read_kb":10408327,"num_write":2629546,"num_write_kb":10518585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80527,"ondisk_log_size":80527,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24464344,"kb_used_data":21002232,"kb_used_omap":1749,"kb_used_meta":3460330,"kb_avail":258651176,"statfs":{"total":289910292480,"available":264858804224,"internally_reserved":0,"allocated":21506285568,"data_stored":25805471181,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1791334,"internal_metadata":3543378586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,4,2,8,13,13,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19886080,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5743,"num_read_kb":22928,"num_write":6327,"num_write_kb":25308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002979"},"pg_stats":[{"pgid":"2.7","version":"160'353254","reported_seq":692504,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:30:05.550514+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:30:05.550514+0000","last_peered":"2023-10-14T01:30:05.550514+0000","last_clean":"2023-10-14T01:30:05.550514+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:30:05.550514+0000","last_undegraded":"2023-10-14T01:30:05.550514+0000","last_fullsized":"2023-10-14T01:30:05.550514+0000","mapping_epoch":13,"log_start":"158'343204","ondisk_log_start":"158'343204","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1240,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 631s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3789349302,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":339545,"num_read_kb":1320347,"num_write":332046,"num_write_kb":1327942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"160'353243","reported_seq":692652,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:30:05.504066+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:30:05.504066+0000","last_peered":"2023-10-14T01:30:05.504066+0000","last_clean":"2023-10-14T01:30:05.504066+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:30:05.504066+0000","last_undegraded":"2023-10-14T01:30:05.504066+0000","last_fullsized":"2023-10-14T01:30:05.504066+0000","mapping_epoch":13,"log_start":"158'343213","ondisk_log_start":"158'343213","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3782533488,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":341267,"num_read_kb":1317277,"num_write":332088,"num_write_kb":1327979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"160'319838","reported_seq":635961,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:30:05.520390+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:30:05.520390+0000","last_peered":"2023-10-14T01:30:05.520390+0000","last_clean":"2023-10-14T01:30:05.520390+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:30:05.520390+0000","last_undegraded":"2023-10-14T01:30:05.520390+0000","last_fullsized":"2023-10-14T01:30:05.520390+0000","mapping_epoch":13,"log_start":"158'309808","ondisk_log_start":"158'309808","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3620098450,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":316296,"num_read_kb":1248880,"num_write":318499,"num_write_kb":1273966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"160'312090","reported_seq":623744,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:30:05.453979+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:30:05.453979+0000","last_peered":"2023-10-14T01:30:05.453979+0000","last_clean":"2023-10-14T01:30:05.453979+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:30:05.453979+0000","last_undegraded":"2023-10-14T01:30:05.453979+0000","last_fullsized":"2023-10-14T01:30:05.453979+0000","mapping_epoch":13,"log_start":"158'302007","ondisk_log_start":"158'302007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3541303427,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":311889,"num_read_kb":1229617,"num_write":310814,"num_write_kb":1243178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"160'351888","reported_seq":703708,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:30:05.092199+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:30:05.092199+0000","last_peered":"2023-10-14T01:30:05.092199+0000","last_clean":"2023-10-14T01:30:05.092199+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:30:05.092199+0000","last_undegraded":"2023-10-14T01:30:05.092199+0000","last_fullsized":"2023-10-14T01:30:05.092199+0000","mapping_epoch":13,"log_start":"158'341809","ondisk_log_start":"158'341809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3983888703,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":352047,"num_read_kb":1382046,"num_write":349301,"num_write_kb":1397057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"160'338764","reported_seq":671650,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:30:04.502829+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:30:04.502829+0000","last_peered":"2023-10-14T01:30:04.502829+0000","last_clean":"2023-10-14T01:30:04.502829+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:30:04.502829+0000","last_undegraded":"2023-10-14T01:30:04.502829+0000","last_fullsized":"2023-10-14T01:30:04.502829+0000","mapping_epoch":13,"log_start":"158'328709","ondisk_log_start":"158'328709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3736162573,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":333135,"num_read_kb":1295588,"num_write":328514,"num_write_kb":1313842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"160'344318","reported_seq":685242,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:30:04.477467+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:30:04.477467+0000","last_peered":"2023-10-14T01:30:04.477467+0000","last_clean":"2023-10-14T01:30:04.477467+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:30:04.477467+0000","last_undegraded":"2023-10-14T01:30:04.477467+0000","last_fullsized":"2023-10-14T01:30:04.477467+0000","mapping_epoch":13,"log_start":"158'334311","ondisk_log_start":"158'334311","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3788914937,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":341164,"num_read_kb":1321488,"num_write":331879,"num_write_kb":1327242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"160'337575","reported_seq":667263,"reported_epoch":160,"state":"active+clean","last_fresh":"2023-10-14T01:30:05.513692+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:30:05.513692+0000","last_peered":"2023-10-14T01:30:05.513692+0000","last_clean":"2023-10-14T01:30:05.513692+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:30:05.513692+0000","last_undegraded":"2023-10-14T01:30:05.513692+0000","last_fullsized":"2023-10-14T01:30:05.513692+0000","mapping_epoch":13,"log_start":"158'327509","ondisk_log_start":"158'327509","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3720364216,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":329883,"num_read_kb":1292975,"num_write":326236,"num_write_kb":1304831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":376,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-14T01:28:17.489789+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:28:17.489789+0000","last_peered":"2023-10-14T01:28:17.489789+0000","last_clean":"2023-10-14T01:28:17.489789+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:28:17.489789+0000","last_undegraded":"2023-10-14T01:28:17.489789+0000","last_fullsized":"2023-10-14T01:28:17.489789+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29962615096,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2665226,"num_read_kb":10408218,"num_write":2629377,"num_write_kb":10516037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21502459904,"data_stored":25803154032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80400,"ondisk_log_size":80400,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739494,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4966076,"kb_used_data":3941044,"kb_used_omap":336,"kb_used_meta":1024687,"kb_avail":89405764,"statfs":{"total":96636764160,"available":91551502336,"internally_reserved":0,"allocated":4035629056,"data_stored":4842657053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":1049280471},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.71699999999999997,"15min":0.82799999999999996},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.319,"5min":5.7560000000000002,"15min":12.356},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.22800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":3.3959999999999999,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.223}]},{"osd":1,"last update":"Sat Oct 14 01:29:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.1639999999999999,"15min":1.2030000000000001},"min":{"1min":0.26400000000000001,"5min":0.22700000000000001,"15min":0.17999999999999999},"max":{"1min":4.0359999999999996,"5min":11.956,"15min":14.005000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1599999999999999,"15min":1.3240000000000001},"min":{"1min":0.22500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":4.2169999999999996,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.36399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739476,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12126756,"kb_used_data":10500636,"kb_used_omap":866,"kb_used_meta":1625245,"kb_avail":82245084,"statfs":{"total":96636764160,"available":84218966016,"internally_reserved":0,"allocated":10752651264,"data_stored":12902661559,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":886933,"internal_metadata":1664251755},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,1,2,8,5,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:50 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.95999999999999996,"15min":0.93799999999999994},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.43099999999999999,"5min":15.122999999999999,"15min":23.434000000000001},"last":5.133},{"interface":"front","average":{"1min":1.429,"5min":1.3640000000000001,"15min":1.288},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":7.8109999999999999,"5min":15.147,"15min":24.908000000000001},"last":0.42099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:29:11 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.1819999999999999,"15min":1.3069999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.8929999999999998,"5min":12.818,"15min":27.837},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.179,"15min":1.304},"min":{"1min":0.219,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":4.4630000000000001,"5min":15.09,"15min":25.710000000000001},"last":0.36799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739491,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7371512,"kb_used_data":6560552,"kb_used_omap":547,"kb_used_meta":810396,"kb_avail":87000328,"statfs":{"total":96636764160,"available":89088335872,"internally_reserved":0,"allocated":6718005248,"data_stored":8060152569,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":560296,"internal_metadata":829846360},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,7,6,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":0.89100000000000001,"15min":0.93799999999999994},"min":{"1min":0.192,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.505000000000001,"5min":10.505000000000001,"15min":11.734999999999999},"last":4.3099999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.64000000000000001,"15min":0.63100000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.17899999999999999},"max":{"1min":10.488,"5min":10.488,"15min":10.488},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.222,"15min":1.0149999999999999},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":6.6079999999999997,"5min":8.8049999999999997,"15min":9.8059999999999992},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.68000000000000005,"15min":0.61699999999999999},"min":{"1min":0.23200000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.9359999999999999,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6716620800,"data_stored":8059106938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10751639552,"data_stored":12901935416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4034199552,"data_stored":4842111678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:30:06.588 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:30:06.588 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:30:06.821 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.817+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:06.825 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.825+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:06.847 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:06.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:06.848 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.849+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:06.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:06.858 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:06.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:06.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:07.143 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:07.144 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:07.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:07.145 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.145+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:07.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:07.159 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:07.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.153+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:07.995 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:07.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:10.741 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3c53f380 2023-10-14T01:30:10.745 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.741+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:30:10.750 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.749+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:10.765 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.765+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:10.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:10.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:10.786 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.785+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:10.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:10.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:10.795 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:10.793+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:12.422 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:12.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.441+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:12.472 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:12.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:12.473 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.473+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:12.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:12.508 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:12.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:12.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:13.292 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.293+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:13.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:13.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:13.306 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.305+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:13.557 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:13.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:13.558 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.553+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:13.570 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:13.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:13.634 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:13.633+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:14.978 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:14.969+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:15.014 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:15.013+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:15.041 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:15.033+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:15.649 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:15.649+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:15.675 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:15.673+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:15.775 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:15.773+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:16.977 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:16.977+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:16.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:16.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:16.987 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:16.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:16.993 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:16.993+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:17.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.861+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:17.867 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.865+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:17.892 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.893+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:17.916 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:17.922 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.921+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:17.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:17.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:17.956 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.957+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:17.980 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:17.981+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:18.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:18.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:18.469 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:18.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:18.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:18.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:18.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:18.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:18.470 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:18.469+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:18.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:18.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:18.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:18.497+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:19.193 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.189+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:19.205 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.205+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:19.211 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:19.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:19.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.213+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:19.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.237+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:19.477 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.477+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:19.509 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.509+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:19.527 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:19.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:19.528 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.525+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:19.569 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.569+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:19.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.617+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:19.627 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.629+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:19.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:19.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:19.635 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.633+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:19.681 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:19.681+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:21.423 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:21.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:21.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:21.417+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:21.426 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:21.421+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:21.437 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:21.437+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:22.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:22.737+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:22.737 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:22.737+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:22.742 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:22.741+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:22.759 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:22.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:22.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:22.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:30:22.760 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:22.757+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:22.936 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:22.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:22.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:22.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:22.937 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:22.937+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:22.945 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:22.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:22.954 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:22.953+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:22.960 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:22.961+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:23.540 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:23.541+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:23.584 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:23.581+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:23.609 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:23.609+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:25.728 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:25.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:25.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:25.729+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:25.753 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:25.753+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:25.769 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:25.769+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:25.787 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:25.781+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:26.422 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:26.421+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:26.590 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:30:27.383 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:30:27.384 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:30:27.429 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2987,"stamp":"2023-10-14T01:30:26.020638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29983931224,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2672164,"num_read_kb":10435559,"num_write":2636501,"num_write_kb":10546405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80657,"ondisk_log_size":80657,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":24757048,"kb_used_data":21043664,"kb_used_omap":1749,"kb_used_meta":3711594,"kb_avail":258358472,"statfs":{"total":289910292480,"available":264559075328,"internally_reserved":0,"allocated":21548711936,"data_stored":25843083534,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1791334,"internal_metadata":3800672922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,7,7,12,10,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14729216,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4808,"num_read_kb":19224,"num_write":4916,"num_write_kb":19664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001996"},"pg_stats":[{"pgid":"2.7","version":"161'354277","reported_seq":694451,"reported_epoch":161,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:30:20.999036+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:30:20.999036+0000","last_peered":"2023-10-14T01:30:20.999036+0000","last_clean":"2023-10-14T01:30:20.999036+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:30:20.999036+0000","last_undegraded":"2023-10-14T01:30:20.999036+0000","last_fullsized":"2023-10-14T01:30:20.999036+0000","mapping_epoch":13,"log_start":"158'344204","ondisk_log_start":"158'344204","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1250,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 646s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3792413110,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":340454,"num_read_kb":1323983,"num_write":332995,"num_write_kb":1331738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"161'354089","reported_seq":694207,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:20.973663+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:30:20.973663+0000","last_peered":"2023-10-14T01:30:20.973663+0000","last_clean":"2023-10-14T01:30:20.973663+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:30:20.973663+0000","last_undegraded":"2023-10-14T01:30:20.973663+0000","last_fullsized":"2023-10-14T01:30:20.973663+0000","mapping_epoch":13,"log_start":"158'344013","ondisk_log_start":"158'344013","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3784622448,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":341961,"num_read_kb":1320053,"num_write":332854,"num_write_kb":1331043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"161'320586","reported_seq":637497,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:20.950014+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:30:20.950014+0000","last_peered":"2023-10-14T01:30:20.950014+0000","last_clean":"2023-10-14T01:30:20.950014+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:30:20.950014+0000","last_undegraded":"2023-10-14T01:30:20.950014+0000","last_fullsized":"2023-10-14T01:30:20.950014+0000","mapping_epoch":13,"log_start":"158'310508","ondisk_log_start":"158'310508","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3622334866,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":317083,"num_read_kb":1252028,"num_write":319247,"num_write_kb":1276958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"161'312878","reported_seq":625292,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:20.969044+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:30:20.969044+0000","last_peered":"2023-10-14T01:30:20.969044+0000","last_clean":"2023-10-14T01:30:20.969044+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:30:20.969044+0000","last_undegraded":"2023-10-14T01:30:20.969044+0000","last_fullsized":"2023-10-14T01:30:20.969044+0000","mapping_epoch":13,"log_start":"158'302807","ondisk_log_start":"158'302807","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3543556227,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":312648,"num_read_kb":1232645,"num_write":311602,"num_write_kb":1246330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"161'353045","reported_seq":705944,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:25.195980+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:30:25.195980+0000","last_peered":"2023-10-14T01:30:25.195980+0000","last_clean":"2023-10-14T01:30:25.195980+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:30:25.195980+0000","last_undegraded":"2023-10-14T01:30:25.195980+0000","last_fullsized":"2023-10-14T01:30:25.195980+0000","mapping_epoch":13,"log_start":"158'343009","ondisk_log_start":"158'343009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3987484991,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":353120,"num_read_kb":1386338,"num_write":350457,"num_write_kb":1401681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"161'339693","reported_seq":673500,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:24.703981+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:30:24.703981+0000","last_peered":"2023-10-14T01:30:24.703981+0000","last_clean":"2023-10-14T01:30:24.703981+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:30:24.703981+0000","last_undegraded":"2023-10-14T01:30:24.703981+0000","last_fullsized":"2023-10-14T01:30:24.703981+0000","mapping_epoch":13,"log_start":"158'329609","ondisk_log_start":"158'329609","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3738693901,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":334047,"num_read_kb":1299236,"num_write":329407,"num_write_kb":1317414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"161'345301","reported_seq":687222,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:24.693343+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:30:24.693343+0000","last_peered":"2023-10-14T01:30:24.693343+0000","last_clean":"2023-10-14T01:30:24.693343+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:30:24.693343+0000","last_undegraded":"2023-10-14T01:30:24.693343+0000","last_fullsized":"2023-10-14T01:30:24.693343+0000","mapping_epoch":13,"log_start":"158'335211","ondisk_log_start":"158'335211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3791806713,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":342144,"num_read_kb":1325400,"num_write":332819,"num_write_kb":1331002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"161'338331","reported_seq":668727,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:20.970221+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:30:20.970221+0000","last_peered":"2023-10-14T01:30:20.970221+0000","last_clean":"2023-10-14T01:30:20.970221+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:30:20.970221+0000","last_undegraded":"2023-10-14T01:30:20.970221+0000","last_fullsized":"2023-10-14T01:30:20.970221+0000","mapping_epoch":13,"log_start":"158'328309","ondisk_log_start":"158'328309","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3722428600,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":330581,"num_read_kb":1295767,"num_write":326951,"num_write_kb":1307691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":378,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:10.068174+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:30:10.068174+0000","last_peered":"2023-10-14T01:30:10.068174+0000","last_clean":"2023-10-14T01:30:10.068174+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:30:10.068174+0000","last_undegraded":"2023-10-14T01:30:10.068174+0000","last_fullsized":"2023-10-14T01:30:10.068174+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":29983340856,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2672038,"num_read_kb":10435450,"num_write":2636332,"num_write_kb":10543857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21545603072,"data_stored":25841435248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80530,"ondisk_log_size":80530,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739498,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040716,"kb_used_data":3949056,"kb_used_omap":336,"kb_used_meta":1091311,"kb_avail":89331124,"statfs":{"total":96636764160,"available":91475070976,"internally_reserved":0,"allocated":4043833344,"data_stored":4849935240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":1117503447},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.71699999999999997,"15min":0.82799999999999996},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.319,"5min":5.7560000000000002,"15min":12.356},"last":1.8380000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.22800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":3.3959999999999999,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.495}]},{"osd":1,"last update":"Sat Oct 14 01:29:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.1639999999999999,"15min":1.2030000000000001},"min":{"1min":0.26400000000000001,"5min":0.22700000000000001,"15min":0.17999999999999999},"max":{"1min":4.0359999999999996,"5min":11.956,"15min":14.005000000000001},"last":0.215},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1599999999999999,"15min":1.3240000000000001},"min":{"1min":0.22500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":4.2169999999999996,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.29599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739479,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12252892,"kb_used_data":10518476,"kb_used_omap":866,"kb_used_meta":1733533,"kb_avail":82118948,"statfs":{"total":96636764160,"available":84089802752,"internally_reserved":0,"allocated":10770919424,"data_stored":12918836258,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":886933,"internal_metadata":1775138667},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,6,8,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:50 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.95999999999999996,"15min":0.93799999999999994},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.43099999999999999,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.3640000000000001,"15min":1.288},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":7.8109999999999999,"5min":15.147,"15min":24.908000000000001},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:30:15 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.153,"15min":1.3089999999999999},"min":{"1min":0.27700000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.4969999999999999,"5min":12.818,"15min":27.837},"last":1.3320000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.206,"15min":1.3240000000000001},"min":{"1min":0.223,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":5.7949999999999999,"5min":15.09,"15min":25.710000000000001},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359739495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7463440,"kb_used_data":6576132,"kb_used_omap":547,"kb_used_meta":886748,"kb_avail":86908400,"statfs":{"total":96636764160,"available":88994201600,"internally_reserved":0,"allocated":6733959168,"data_stored":8074312036,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":560296,"internal_metadata":908030808},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":0.89100000000000001,"15min":0.93799999999999994},"min":{"1min":0.192,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.505000000000001,"5min":10.505000000000001,"15min":11.734999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.64000000000000001,"15min":0.63100000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.17899999999999999},"max":{"1min":10.488,"5min":10.488,"15min":10.488},"last":0.247}]},{"osd":2,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.222,"15min":1.0149999999999999},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":6.6079999999999997,"5min":8.8049999999999997,"15min":9.8059999999999992},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.68000000000000005,"15min":0.61699999999999999},"min":{"1min":0.23200000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.9359999999999999,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6732754944,"data_stored":8073438842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10770329600,"data_stored":12918503736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4042518528,"data_stored":4849492670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:30:27.430 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:30:27.431 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:30:30.859 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:30.857+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:30.918 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:30.917+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:30.941 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:30.941+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:31.001 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:31.001+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:31.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:31.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:30:31.793 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:31.792+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:31.822 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:31.820+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:32.261 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.260+0000 7feb91e47640 -1 reset not still connected to 0x5632e2071930 2023-10-14T01:30:32.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:30:32.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:32.297 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.288+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:32.335 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.336+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:32.358 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:32.358 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:30:32.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:32.352+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:33.615 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:33.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:33.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:33.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:33.616 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:33.612+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:33.652 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:33.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:33.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:33.660+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:34.863 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:34.860+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:34.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:34.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:34.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:34.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:34.896 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:34.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:34.910 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:34.908+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:34.912 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:34.912+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:34.921 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:34.920+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:35.596 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:35.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:30:35.597 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:35.596+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:35.618 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:35.616+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:35.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:35.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:35.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:35.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:35.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:35.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:30:35.621 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:35.620+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:36.170 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:36.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:36.171 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.164+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:36.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.172+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:36.179 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.176+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:36.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:36.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:36.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:36.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:37.444 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:37.444+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:37.457 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:37.452+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:37.502 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:37.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:37.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:37.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:37.503 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:37.500+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:38.705 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:38.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:38.709 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:38.708+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:39.438 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:39.436+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:39.441 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:39.440+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:39.450 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:39.448+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:41.645 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:41.644+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:41.667 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:41.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:41.686 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:41.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3d82fa00 2023-10-14T01:30:41.738 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:41.736+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:41.770 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:41.768+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:41.785 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:41.784+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:41.791 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:41.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:42.313 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.312+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:42.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:42.326 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:42.327 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.324+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:42.391 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.388+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:42.399 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.396+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:42.566 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.564+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:42.633 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:42.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:42.634 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.632+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:42.658 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.652+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:42.670 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.668+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:42.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:42.680+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:43.141 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:43.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:43.798 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:43.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:43.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:43.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:43.799 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.796+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:43.806 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:43.808 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.804+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:43.828 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:43.828+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:44.895 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:44.896+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:44.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:44.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:44.907 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:44.904+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:44.930 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:44.924+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:44.944 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:44.944+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:44.953 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:44.952+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:47.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:47.038 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.028+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:47.050 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:47.216 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:30:47.217 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.216+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:47.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:47.226 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:47.227 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.224+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:47.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:47.228+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:47.432 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:30:48.199 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:48.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:48.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:48.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:48.200 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:48.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:48.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:48.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:48.227 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:30:48.228 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:30:48.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:48.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:48.236 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:48.236+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:48.281 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":2998,"stamp":"2023-10-14T01:30:48.024611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30011276120,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2681522,"num_read_kb":10472947,"num_write":2646058,"num_write_kb":10584633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80582,"ondisk_log_size":80582,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25049424,"kb_used_data":21096176,"kb_used_omap":1749,"kb_used_meta":3951466,"kb_avail":258066096,"statfs":{"total":289910292480,"available":264259682304,"internally_reserved":0,"allocated":21602484224,"data_stored":25890339086,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1791334,"internal_metadata":4046301850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,3,11,15,18,10,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15089664,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5326,"num_read_kb":21284,"num_write":5384,"num_write_kb":21536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001962"},"pg_stats":[{"pgid":"2.7","version":"161'355753","reported_seq":697301,"reported_epoch":161,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:30:46.691271+0000","last_change":"2023-10-14T01:19:34.249633+0000","last_active":"2023-10-14T01:30:46.691271+0000","last_peered":"2023-10-14T01:30:46.691271+0000","last_clean":"2023-10-14T01:30:46.691271+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:30:46.691271+0000","last_undegraded":"2023-10-14T01:30:46.691271+0000","last_fullsized":"2023-10-14T01:30:46.691271+0000","mapping_epoch":13,"log_start":"159'345704","ondisk_log_start":"159'345704","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-13T23:56:13.728341+0000","last_clean_scrub_stamp":"2023-10-13T23:56:13.728341+0000","objects_scrubbed":1297,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 672s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3796316598,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":341794,"num_read_kb":1329335,"num_write":334365,"num_write_kb":1337218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"161'355512","reported_seq":696900,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:46.645976+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:30:46.645976+0000","last_peered":"2023-10-14T01:30:46.645976+0000","last_clean":"2023-10-14T01:30:46.645976+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:30:46.645976+0000","last_undegraded":"2023-10-14T01:30:46.645976+0000","last_fullsized":"2023-10-14T01:30:46.645976+0000","mapping_epoch":13,"log_start":"159'345413","ondisk_log_start":"159'345413","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3788587376,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":343210,"num_read_kb":1325037,"num_write":334202,"num_write_kb":1336435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"161'321846","reported_seq":640004,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:46.675260+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:30:46.675260+0000","last_peered":"2023-10-14T01:30:46.675260+0000","last_clean":"2023-10-14T01:30:46.675260+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:30:46.675260+0000","last_undegraded":"2023-10-14T01:30:46.675260+0000","last_fullsized":"2023-10-14T01:30:46.675260+0000","mapping_epoch":13,"log_start":"158'311808","ondisk_log_start":"158'311808","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3625824658,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":318334,"num_read_kb":1257028,"num_write":320507,"num_write_kb":1281998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"161'314051","reported_seq":627650,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:46.654841+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:30:46.654841+0000","last_peered":"2023-10-14T01:30:46.654841+0000","last_clean":"2023-10-14T01:30:46.654841+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:30:46.654841+0000","last_undegraded":"2023-10-14T01:30:46.654841+0000","last_fullsized":"2023-10-14T01:30:46.654841+0000","mapping_epoch":13,"log_start":"158'304007","ondisk_log_start":"158'304007","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3546747011,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":313833,"num_read_kb":1237385,"num_write":312775,"num_write_kb":1251022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"161'354082","reported_seq":708021,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:45.407306+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:30:45.407306+0000","last_peered":"2023-10-14T01:30:45.407306+0000","last_clean":"2023-10-14T01:30:45.407306+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:30:45.407306+0000","last_undegraded":"2023-10-14T01:30:45.407306+0000","last_fullsized":"2023-10-14T01:30:45.407306+0000","mapping_epoch":13,"log_start":"158'344009","ondisk_log_start":"158'344009","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3990348095,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":354158,"num_read_kb":1390486,"num_write":351491,"num_write_kb":1405817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"161'340725","reported_seq":675550,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:44.966995+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:30:44.966995+0000","last_peered":"2023-10-14T01:30:44.966995+0000","last_clean":"2023-10-14T01:30:44.966995+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:30:44.966995+0000","last_undegraded":"2023-10-14T01:30:44.966995+0000","last_fullsized":"2023-10-14T01:30:44.966995+0000","mapping_epoch":13,"log_start":"158'330709","ondisk_log_start":"158'330709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3741593869,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":335047,"num_read_kb":1303232,"num_write":330419,"num_write_kb":1321462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"161'346287","reported_seq":689155,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:44.941887+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:30:44.941887+0000","last_peered":"2023-10-14T01:30:44.941887+0000","last_clean":"2023-10-14T01:30:44.941887+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:30:44.941887+0000","last_undegraded":"2023-10-14T01:30:44.941887+0000","last_fullsized":"2023-10-14T01:30:44.941887+0000","mapping_epoch":13,"log_start":"158'336211","ondisk_log_start":"158'336211","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3794579705,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":343068,"num_read_kb":1329092,"num_write":333778,"num_write_kb":1334838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"161'339769","reported_seq":671549,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:46.601849+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:30:46.601849+0000","last_peered":"2023-10-14T01:30:46.601849+0000","last_clean":"2023-10-14T01:30:46.601849+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:30:46.601849+0000","last_undegraded":"2023-10-14T01:30:46.601849+0000","last_fullsized":"2023-10-14T01:30:46.601849+0000","mapping_epoch":13,"log_start":"159'329709","ondisk_log_start":"159'329709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3726688440,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":331952,"num_read_kb":1301243,"num_write":328352,"num_write_kb":1313295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":378,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:10.068174+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:30:10.068174+0000","last_peered":"2023-10-14T01:30:10.068174+0000","last_clean":"2023-10-14T01:30:10.068174+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:30:10.068174+0000","last_undegraded":"2023-10-14T01:30:10.068174+0000","last_fullsized":"2023-10-14T01:30:10.068174+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":30010685752,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2681396,"num_read_kb":10472838,"num_write":2645889,"num_write_kb":10582085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":11428,"num_omap_keys":263,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21598920704,"data_stored":25888248432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80455,"ondisk_log_size":80455,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739502,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5081436,"kb_used_data":3957840,"kb_used_omap":336,"kb_used_meta":1123247,"kb_avail":89290404,"statfs":{"total":96636764160,"available":91433373696,"internally_reserved":0,"allocated":4052828160,"data_stored":4857738120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":1150205911},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.71699999999999997,"15min":0.82799999999999996},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.319,"5min":5.7560000000000002,"15min":12.356},"last":0.252},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.22800000000000001,"5min":0.191,"15min":0.191},"max":{"1min":3.3959999999999999,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.215}]},{"osd":1,"last update":"Sat Oct 14 01:29:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.1639999999999999,"15min":1.2030000000000001},"min":{"1min":0.26400000000000001,"5min":0.22700000000000001,"15min":0.17999999999999999},"max":{"1min":4.0359999999999996,"5min":11.956,"15min":14.005000000000001},"last":0.443},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1599999999999999,"15min":1.3240000000000001},"min":{"1min":0.22500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":4.2169999999999996,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.27300000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739484,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12423004,"kb_used_data":10548372,"kb_used_omap":866,"kb_used_meta":1873757,"kb_avail":81948836,"statfs":{"total":96636764160,"available":83915608064,"internally_reserved":0,"allocated":10801532928,"data_stored":12945779746,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":886933,"internal_metadata":1918728043},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,8,4,8,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:50 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.95999999999999996,"15min":0.93799999999999994},"min":{"1min":0.19700000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.43099999999999999,"5min":15.122999999999999,"15min":23.434000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.3640000000000001,"15min":1.288},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":7.8109999999999999,"5min":15.147,"15min":24.908000000000001},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:30:15 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.153,"15min":1.3089999999999999},"min":{"1min":0.27700000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.4969999999999999,"5min":12.818,"15min":27.837},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.206,"15min":1.3240000000000001},"min":{"1min":0.223,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":5.7949999999999999,"5min":15.09,"15min":25.710000000000001},"last":0.253}]}]},{"osd":0,"up_from":8,"seq":34359739499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7544984,"kb_used_data":6589964,"kb_used_omap":547,"kb_used_meta":954460,"kb_avail":86826856,"statfs":{"total":96636764160,"available":88910700544,"internally_reserved":0,"allocated":6748123136,"data_stored":8086821220,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":560296,"internal_metadata":977367896},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":0.89100000000000001,"15min":0.93799999999999994},"min":{"1min":0.192,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.505000000000001,"5min":10.505000000000001,"15min":11.734999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.64000000000000001,"15min":0.63100000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.17899999999999999},"max":{"1min":10.488,"5min":10.488,"15min":10.488},"last":0.23699999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.222,"15min":1.0149999999999999},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":6.6079999999999997,"5min":8.8049999999999997,"15min":9.8059999999999992},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.68000000000000005,"15min":0.61699999999999999},"min":{"1min":0.23200000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.9359999999999999,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.51200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6747021312,"data_stored":8085993082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10800259072,"data_stored":12944841016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4051640320,"data_stored":4857414334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:30:48.283 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-13T23:56:13.728341+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=13, tm_hour=23, tm_min=56, tm_sec=13, tm_wday=4, tm_yday=286, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=1, tm_sec=50, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:30:48.284 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:30:50.346 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:50.344+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:30:50.356 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:50.356+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:30:50.359 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:50.356+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:30:50.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:50.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:50.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:50.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:30:50.364 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:50.364+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:50.577 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:50.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:30:50.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:50.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:50.582 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:50.580+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:50.594 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:50.592+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:50.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:50.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:50.625 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:50.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:54.134 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:54.132+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:54.152 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:54.152+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:56.072 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:56.072+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:56.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:56.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:56.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:56.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:56.079 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:56.076+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:57.419 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:57.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:30:57.420 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:30:57.420+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:30:57.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:57.479 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.472+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:57.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:57.480 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.480+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:30:57.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.684+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:30:57.706 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:30:57.707 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.704+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:30:57.721 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.720+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:30:57.729 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:30:57.742 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:30:57.740+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:00.069 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:00.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:00.090 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:00.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:00.091 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:00.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:00.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:00.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:00.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:00.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:00.109 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:00.108+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:01.461 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:01.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:31:01.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:01.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:01.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:01.460+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:01.465 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:01.464+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:01.492 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:01.492+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:01.543 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:01.544+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:02.171 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:02.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:31:02.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:02.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:31:02.172 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:02.168+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:31:02.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:02.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:02.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:02.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:02.931 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:02.928+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:02.967 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:02.964+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:03.023 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.020+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:31:03.034 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.032+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:03.044 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.044+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:03.057 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.056+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:03.201 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.200+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:31:03.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:03.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:31:03.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:03.208 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:03.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:03.209 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:03.204+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:04.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.148+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:04.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:31:04.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:04.160 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.160+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:04.169 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:04.186 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.184+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:04.195 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:04.612 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.608+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:04.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:04.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:05.068 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.068+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:31:05.086 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.080+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:05.148 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:31:05.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:05.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:05.149 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.140+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:05.174 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.168+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:05.180 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:05.180+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:05.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:05.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e20e7110 2023-10-14T01:31:05.209 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:05.208+0000 7feb91e47640 -1 reset not still connected to 0x5632e260e680 2023-10-14T01:31:05.211 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:05.212+0000 7feb91e47640 -1 reset not still connected to 0x5632e26521a0 2023-10-14T01:31:05.255 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:05.252+0000 7feb91e47640 -1 reset not still connected to 0x5632e2653520 2023-10-14T01:31:05.278 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:05.276+0000 7feb91e47640 -1 reset not still connected to 0x5632e27cb5f0 2023-10-14T01:31:05.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:05.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e289f520 2023-10-14T01:31:05.284 INFO:tasks.ceph.osd.2.smithi138.stderr:2023-10-14T01:31:05.284+0000 7feb91e47640 -1 reset not still connected to 0x5632e29976c0 2023-10-14T01:31:06.093 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.088+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:06.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:06.196 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:06.197 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.196+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:06.572 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.572+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:06.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:31:06.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:06.599 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:06.600 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.596+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:06.623 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:06.624 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:06.624+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:07.730 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:07.728+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:07.752 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:07.752+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:07.757 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:07.756+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:07.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:07.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:07.789 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:07.788+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:08.285 DEBUG:teuthology.orchestra.run.smithi138:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:31:09.045 INFO:teuthology.orchestra.run.smithi138.stdout: 2023-10-14T01:31:09.047 INFO:teuthology.orchestra.run.smithi138.stderr:dumped all 2023-10-14T01:31:09.109 INFO:teuthology.orchestra.run.smithi138.stdout:{"pg_ready":true,"pg_map":{"version":3008,"stamp":"2023-10-14T01:31:08.031073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30032640856,"num_objects":10326,"num_object_clones":5120,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10326,"num_whiteouts":0,"num_read":2689257,"num_read_kb":10503867,"num_write":2654077,"num_write_kb":10616709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13460,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80535,"ondisk_log_size":80535,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":25464636,"kb_used_data":21137432,"kb_used_omap":1749,"kb_used_meta":4325418,"kb_avail":257650884,"statfs":{"total":289910292480,"available":263834505216,"internally_reserved":0,"allocated":21644730368,"data_stored":25926174990,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":1791334,"internal_metadata":4429228698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,5,14,10,31,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13586432,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5048,"num_read_kb":20180,"num_write":5244,"num_write_kb":20976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004527"},"pg_stats":[{"pgid":"2.7","version":"161'356861","reported_seq":699407,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:07.037257+0000","last_change":"2023-10-14T01:30:53.826799+0000","last_active":"2023-10-14T01:31:07.037257+0000","last_peered":"2023-10-14T01:31:07.037257+0000","last_clean":"2023-10-14T01:31:07.037257+0000","last_became_active":"2023-10-13T23:56:14.756964+0000","last_became_peered":"2023-10-13T23:56:14.756964+0000","last_unstale":"2023-10-14T01:31:07.037257+0000","last_undegraded":"2023-10-14T01:31:07.037257+0000","last_fullsized":"2023-10-14T01:31:07.037257+0000","mapping_epoch":13,"log_start":"159'346804","ondisk_log_start":"159'346804","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"161'356143","last_scrub_stamp":"2023-10-14T01:30:53.826618+0000","last_deep_scrub":"161'356143","last_deep_scrub_stamp":"2023-10-14T01:30:53.826618+0000","last_clean_scrub_stamp":"2023-10-14T01:30:53.826618+0000","objects_scrubbed":1310,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":680,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:38:00.022532+0000","scrub_duration":679.57685386699995,"objects_trimmed":0,"snaptrim_duration":0.0031894850000000001,"stat_sum":{"num_bytes":3798983094,"num_objects":1310,"num_object_clones":648,"num_object_copies":2620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":342767,"num_read_kb":1333223,"num_write":335391,"num_write_kb":1341322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"161'356571","reported_seq":698882,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:07.039298+0000","last_change":"2023-10-14T01:21:46.951575+0000","last_active":"2023-10-14T01:31:07.039298+0000","last_peered":"2023-10-14T01:31:07.039298+0000","last_clean":"2023-10-14T01:31:07.039298+0000","last_became_active":"2023-10-13T23:56:14.755669+0000","last_became_peered":"2023-10-13T23:56:14.755669+0000","last_unstale":"2023-10-14T01:31:07.039298+0000","last_undegraded":"2023-10-14T01:31:07.039298+0000","last_fullsized":"2023-10-14T01:31:07.039298+0000","mapping_epoch":13,"log_start":"159'346513","ondisk_log_start":"159'346513","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'325457","last_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_deep_scrub":"153'325457","last_deep_scrub_stamp":"2023-10-14T01:21:46.951342+0000","last_clean_scrub_stamp":"2023-10-14T01:21:46.951342+0000","objects_scrubbed":1305,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":679,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:31:31.115489+0000","scrub_duration":678.84059137600002,"objects_trimmed":0,"snaptrim_duration":0.00310122,"stat_sum":{"num_bytes":3791241584,"num_objects":1305,"num_object_clones":646,"num_object_copies":2610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":344119,"num_read_kb":1328673,"num_write":335180,"num_write_kb":1340347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3442,"num_omap_keys":83,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"161'322753","reported_seq":641830,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:06.907091+0000","last_change":"2023-10-14T01:27:18.080805+0000","last_active":"2023-10-14T01:31:06.907091+0000","last_peered":"2023-10-14T01:31:06.907091+0000","last_clean":"2023-10-14T01:31:06.907091+0000","last_became_active":"2023-10-13T23:56:14.756844+0000","last_became_peered":"2023-10-13T23:56:14.756844+0000","last_unstale":"2023-10-14T01:31:06.907091+0000","last_undegraded":"2023-10-14T01:31:06.907091+0000","last_fullsized":"2023-10-14T01:31:06.907091+0000","mapping_epoch":13,"log_start":"159'312708","ondisk_log_start":"159'312708","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'311080","last_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_deep_scrub":"158'311080","last_deep_scrub_stamp":"2023-10-14T01:27:18.080561+0000","last_clean_scrub_stamp":"2023-10-14T01:27:18.080561+0000","objects_scrubbed":1246,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":477,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:28:48.909008+0000","scrub_duration":476.93586161000002,"objects_trimmed":0,"snaptrim_duration":0.0037208179999999999,"stat_sum":{"num_bytes":3628224914,"num_objects":1246,"num_object_clones":618,"num_object_copies":2492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":319249,"num_read_kb":1260684,"num_write":321414,"num_write_kb":1285626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"161'314982","reported_seq":629492,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:07.003153+0000","last_change":"2023-10-14T01:19:33.425881+0000","last_active":"2023-10-14T01:31:07.003153+0000","last_peered":"2023-10-14T01:31:07.003153+0000","last_clean":"2023-10-14T01:31:07.003153+0000","last_became_active":"2023-10-13T23:56:14.754685+0000","last_became_peered":"2023-10-13T23:56:14.754685+0000","last_unstale":"2023-10-14T01:31:07.003153+0000","last_undegraded":"2023-10-14T01:31:07.003153+0000","last_fullsized":"2023-10-14T01:31:07.003153+0000","mapping_epoch":13,"log_start":"159'304907","ondisk_log_start":"159'304907","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'281284","last_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_deep_scrub":"150'281284","last_deep_scrub_stamp":"2023-10-14T01:19:33.424244+0000","last_clean_scrub_stamp":"2023-10-14T01:19:33.424244+0000","objects_scrubbed":1217,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:06:32.833282+0000","scrub_duration":424.29815268700003,"objects_trimmed":0,"snaptrim_duration":0.0025071949999999998,"stat_sum":{"num_bytes":3549151363,"num_objects":1217,"num_object_clones":605,"num_object_copies":2434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":314744,"num_read_kb":1241025,"num_write":313706,"num_write_kb":1254746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":530,"num_omap_keys":40,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"161'355185","reported_seq":710188,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:05.723231+0000","last_change":"2023-10-14T01:12:27.820066+0000","last_active":"2023-10-14T01:31:05.723231+0000","last_peered":"2023-10-14T01:31:05.723231+0000","last_clean":"2023-10-14T01:31:05.723231+0000","last_became_active":"2023-10-13T23:56:14.753122+0000","last_became_peered":"2023-10-13T23:56:14.753122+0000","last_unstale":"2023-10-14T01:31:05.723231+0000","last_undegraded":"2023-10-14T01:31:05.723231+0000","last_fullsized":"2023-10-14T01:31:05.723231+0000","mapping_epoch":13,"log_start":"159'345109","ondisk_log_start":"159'345109","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'295072","last_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_deep_scrub":"143'295072","last_deep_scrub_stamp":"2023-10-14T01:12:27.819920+0000","last_clean_scrub_stamp":"2023-10-14T01:12:27.819920+0000","objects_scrubbed":1373,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":634,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:26:26.871114+0000","scrub_duration":633.21803747399997,"objects_trimmed":0,"snaptrim_duration":0.0010636389999999999,"stat_sum":{"num_bytes":3993309503,"num_objects":1373,"num_object_clones":681,"num_object_copies":2746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":355215,"num_read_kb":1394714,"num_write":352590,"num_write_kb":1410213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1868,"num_omap_keys":30,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"161'341772","reported_seq":677634,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:05.103171+0000","last_change":"2023-10-14T01:10:19.532572+0000","last_active":"2023-10-14T01:31:05.103171+0000","last_peered":"2023-10-14T01:31:05.103171+0000","last_clean":"2023-10-14T01:31:05.103171+0000","last_became_active":"2023-10-13T23:56:14.754374+0000","last_became_peered":"2023-10-13T23:56:14.754374+0000","last_unstale":"2023-10-14T01:31:05.103171+0000","last_undegraded":"2023-10-14T01:31:05.103171+0000","last_fullsized":"2023-10-14T01:31:05.103171+0000","mapping_epoch":13,"log_start":"159'331709","ondisk_log_start":"159'331709","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'278003","last_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_deep_scrub":"141'278003","last_deep_scrub_stamp":"2023-10-14T01:10:19.532384+0000","last_clean_scrub_stamp":"2023-10-14T01:10:19.532384+0000","objects_scrubbed":1286,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":500,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:08:28.500632+0000","scrub_duration":499.48607145699998,"objects_trimmed":0,"snaptrim_duration":0.000870323,"stat_sum":{"num_bytes":3744461069,"num_objects":1286,"num_object_clones":638,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":336068,"num_read_kb":1307316,"num_write":331454,"num_write_kb":1325602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2032,"num_omap_keys":40,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"161'347332","reported_seq":691234,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:05.116272+0000","last_change":"2023-10-14T01:10:24.234716+0000","last_active":"2023-10-14T01:31:05.116272+0000","last_peered":"2023-10-14T01:31:05.116272+0000","last_clean":"2023-10-14T01:31:05.116272+0000","last_became_active":"2023-10-13T23:56:14.754384+0000","last_became_peered":"2023-10-13T23:56:14.754384+0000","last_unstale":"2023-10-14T01:31:05.116272+0000","last_undegraded":"2023-10-14T01:31:05.116272+0000","last_fullsized":"2023-10-14T01:31:05.116272+0000","mapping_epoch":13,"log_start":"159'337311","ondisk_log_start":"159'337311","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'282376","last_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_deep_scrub":"141'282376","last_deep_scrub_stamp":"2023-10-14T01:10:24.234544+0000","last_clean_scrub_stamp":"2023-10-14T01:10:24.234544+0000","objects_scrubbed":1308,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":509,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:57:06.026820+0000","scrub_duration":508.03620039399999,"objects_trimmed":0,"snaptrim_duration":0.0011652069999999999,"stat_sum":{"num_bytes":3797274873,"num_objects":1308,"num_object_clones":648,"num_object_copies":2616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":344085,"num_read_kb":1333160,"num_write":334808,"num_write_kb":1338958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2540,"num_omap_keys":50,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"161'340822","reported_seq":673543,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:31:06.999698+0000","last_change":"2023-10-14T01:19:18.370883+0000","last_active":"2023-10-14T01:31:06.999698+0000","last_peered":"2023-10-14T01:31:06.999698+0000","last_clean":"2023-10-14T01:31:06.999698+0000","last_became_active":"2023-10-13T23:56:14.753620+0000","last_became_peered":"2023-10-13T23:56:14.753620+0000","last_unstale":"2023-10-14T01:31:06.999698+0000","last_undegraded":"2023-10-14T01:31:06.999698+0000","last_fullsized":"2023-10-14T01:31:06.999698+0000","mapping_epoch":13,"log_start":"159'330809","ondisk_log_start":"159'330809","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'303978","last_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_deep_scrub":"150'303978","last_deep_scrub_stamp":"2023-10-14T01:19:18.370715+0000","last_clean_scrub_stamp":"2023-10-14T01:19:18.370715+0000","objects_scrubbed":1279,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":538,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:06:39.021921+0000","scrub_duration":537.23372454000003,"objects_trimmed":0,"snaptrim_duration":0.0024538210000000001,"stat_sum":{"num_bytes":3729404088,"num_objects":1279,"num_object_clones":636,"num_object_copies":2558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":332884,"num_read_kb":1304963,"num_write":329365,"num_write_kb":1317347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1016,"num_omap_keys":20,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"133'127","reported_seq":378,"reported_epoch":161,"state":"active+clean","last_fresh":"2023-10-14T01:30:10.068174+0000","last_change":"2023-10-14T01:01:58.396639+0000","last_active":"2023-10-14T01:30:10.068174+0000","last_peered":"2023-10-14T01:30:10.068174+0000","last_clean":"2023-10-14T01:30:10.068174+0000","last_became_active":"2023-10-13T23:56:09.868842+0000","last_became_peered":"2023-10-13T23:56:09.868842+0000","last_unstale":"2023-10-14T01:30:10.068174+0000","last_undegraded":"2023-10-14T01:30:10.068174+0000","last_fullsized":"2023-10-14T01:30:10.068174+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'127","last_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_deep_scrub":"133'127","last_deep_scrub_stamp":"2023-10-14T01:01:58.396411+0000","last_clean_scrub_stamp":"2023-10-14T01:01:58.396411+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:42:18.911869+0000","scrub_duration":0.067084365000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":30032050488,"num_objects":10324,"num_object_clones":5120,"num_object_copies":20648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10324,"num_whiteouts":0,"num_read":2689131,"num_read_kb":10503758,"num_write":2653908,"num_write_kb":10614161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13460,"num_omap_keys":303,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21641650176,"data_stored":25924457072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33555,"internal_metadata":0},"log_size":80408,"ondisk_log_size":80408,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":12876,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739506,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5139788,"kb_used_data":3966212,"kb_used_omap":336,"kb_used_meta":1173231,"kb_avail":89232052,"statfs":{"total":96636764160,"available":91373621248,"internally_reserved":0,"allocated":4061401088,"data_stored":4865098632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":344105,"internal_metadata":1201389527},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:47 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.69599999999999995,"15min":0.82699999999999996},"min":{"1min":0.20000000000000001,"5min":0.184,"15min":0.184},"max":{"1min":4.8810000000000002,"5min":4.8810000000000002,"15min":12.356},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.58599999999999997,"15min":0.61799999999999999},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.191},"max":{"1min":4.5709999999999997,"5min":5.3579999999999997,"15min":7.6680000000000001},"last":0.216}]},{"osd":1,"last update":"Sat Oct 14 01:30:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.92500000000000004,"15min":1.1719999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.17999999999999999},"max":{"1min":1.6379999999999999,"5min":11.956,"15min":14.005000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.95699999999999996,"15min":1.304},"min":{"1min":0.26500000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.923,"5min":13.342000000000001,"15min":13.342000000000001},"last":0.46000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739488,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":12686524,"kb_used_data":10568620,"kb_used_omap":866,"kb_used_meta":2117021,"kb_avail":81685316,"statfs":{"total":96636764160,"available":83645763584,"internally_reserved":0,"allocated":10822266880,"data_stored":12963310626,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":886933,"internal_metadata":2167830379},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,8,1,25,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:30:53 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.88400000000000001,"15min":0.84299999999999997},"min":{"1min":0.18099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.133,"5min":15.122999999999999,"15min":15.122999999999999},"last":0.187},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.306,"15min":1.069},"min":{"1min":0.221,"5min":0.221,"15min":0.217},"max":{"1min":2.1720000000000002,"5min":15.147,"15min":15.147},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:30:15 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.153,"15min":1.3089999999999999},"min":{"1min":0.27700000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.4969999999999999,"5min":12.818,"15min":27.837},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.206,"15min":1.3240000000000001},"min":{"1min":0.223,"5min":0.20799999999999999,"15min":0.16800000000000001},"max":{"1min":5.7949999999999999,"5min":15.09,"15min":25.710000000000001},"last":0.35599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7638324,"kb_used_data":6602600,"kb_used_omap":547,"kb_used_meta":1035164,"kb_avail":86733516,"statfs":{"total":96636764160,"available":88815120384,"internally_reserved":0,"allocated":6761062400,"data_stored":8097765732,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":560296,"internal_metadata":1060008792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:30:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":0.877,"15min":1},"min":{"1min":0.183,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.6820000000000004,"5min":10.505000000000001,"15min":11.734999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.65400000000000003,"15min":0.64000000000000001},"min":{"1min":0.23699999999999999,"5min":0.20899999999999999,"15min":0.17899999999999999},"max":{"1min":4.3319999999999999,"5min":10.488,"15min":10.488},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:30:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.2709999999999999,"15min":1.0249999999999999},"min":{"1min":0.221,"5min":0.21199999999999999,"15min":0.19400000000000001},"max":{"1min":8.9849999999999994,"5min":8.9849999999999994,"15min":9.8059999999999992},"last":0.312},{"interface":"front","average":{"1min":0.434,"5min":0.58999999999999997,"15min":0.60299999999999998},"min":{"1min":0.255,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.3720000000000001,"5min":8.7729999999999997,"15min":8.7729999999999997},"last":0.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6438,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6760079360,"data_stored":8097035898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10189,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10821648384,"data_stored":12962924856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15008,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4059922432,"data_stored":4864496318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8358,"internal_metadata":0}]}} 2023-10-14T01:31:09.111 DEBUG:teuthology.orchestra.run.smithi138:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-14T01:31:09.212 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d83ec30 2023-10-14T01:31:09.214 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.212+0000 7f23d422d640 -1 reset not still connected to 0x55af3d858750 2023-10-14T01:31:09.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d867860 2023-10-14T01:31:09.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3d8a1450 2023-10-14T01:31:09.232 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3de73380 2023-10-14T01:31:09.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.232+0000 7f23d422d640 -1 reset not still connected to 0x55af3e103930 2023-10-14T01:31:09.246 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.244+0000 7f23d422d640 -1 reset not still connected to 0x55af3e5f72b0 2023-10-14T01:31:09.264 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:09.264+0000 7f23d422d640 -1 reset not still connected to 0x55af3e953930 2023-10-14T01:31:09.937 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-14T01:31:09.938 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-14T01:31:09.938 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-14T01:31:09.939 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:31:11.053 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-14T01:31:13.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.232+0000 7f23d422d640 -1 reset not still connected to 0x55af5026a000 2023-10-14T01:31:13.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.232+0000 7f23d422d640 -1 reset not still connected to 0x55af5ef20270 2023-10-14T01:31:13.268 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.268+0000 7f23d422d640 -1 reset not still connected to 0x55af65e551e0 2023-10-14T01:31:13.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.280+0000 7f23d422d640 -1 reset not still connected to 0x55af66f26820 2023-10-14T01:31:13.282 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.280+0000 7f23d422d640 -1 reset not still connected to 0x55af66f27450 2023-10-14T01:31:13.283 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.284+0000 7f23d422d640 -1 reset not still connected to 0x55af7ca2c820 2023-10-14T01:31:13.305 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.304+0000 7f23d422d640 -1 reset not still connected to 0x55af85f80a90 2023-10-14T01:31:13.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.536+0000 7f23d422d640 -1 reset not still connected to 0x55af5026a000 2023-10-14T01:31:13.537 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.536+0000 7f23d422d640 -1 reset not still connected to 0x55af5ef20270 2023-10-14T01:31:13.563 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.564+0000 7f23d422d640 -1 reset not still connected to 0x55af65e551e0 2023-10-14T01:31:13.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.564+0000 7f23d422d640 -1 reset not still connected to 0x55af66f26820 2023-10-14T01:31:13.564 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.564+0000 7f23d422d640 -1 reset not still connected to 0x55af66f27450 2023-10-14T01:31:13.580 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.580+0000 7f23d422d640 -1 reset not still connected to 0x55af7ca2c820 2023-10-14T01:31:13.591 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.588+0000 7f23d422d640 -1 reset not still connected to 0x55af85f80a90 2023-10-14T01:31:13.641 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.640+0000 7f23d422d640 -1 reset not still connected to 0x55af40c721a0 2023-10-14T01:31:13.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.664+0000 7f23d422d640 -1 reset not still connected to 0x55af5026a000 2023-10-14T01:31:13.666 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.664+0000 7f23d422d640 -1 reset not still connected to 0x55af5ef20270 2023-10-14T01:31:13.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.680+0000 7f23d422d640 -1 reset not still connected to 0x55af65e551e0 2023-10-14T01:31:13.682 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.680+0000 7f23d422d640 -1 reset not still connected to 0x55af66f26820 2023-10-14T01:31:13.683 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.680+0000 7f23d422d640 -1 reset not still connected to 0x55af66f27450 2023-10-14T01:31:13.687 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.688+0000 7f23d422d640 -1 reset not still connected to 0x55af7ca2c820 2023-10-14T01:31:13.715 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:13.712+0000 7f23d422d640 -1 reset not still connected to 0x55af85f80a90 2023-10-14T01:31:14.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:14.488+0000 7f23d422d640 -1 reset not still connected to 0x55af5026a000 2023-10-14T01:31:14.490 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:14.488+0000 7f23d422d640 -1 reset not still connected to 0x55af5ef20270 2023-10-14T01:31:14.520 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:14.520+0000 7f23d422d640 -1 reset not still connected to 0x55af65e551e0 2023-10-14T01:31:14.526 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:14.524+0000 7f23d422d640 -1 reset not still connected to 0x55af66f26820 2023-10-14T01:31:14.533 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:14.532+0000 7f23d422d640 -1 reset not still connected to 0x55af66f27450 2023-10-14T01:31:14.541 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:14.540+0000 7f23d422d640 -1 reset not still connected to 0x55af7ca2c820 2023-10-14T01:31:14.562 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:14.560+0000 7f23d422d640 -1 reset not still connected to 0x55af85f80a90 2023-10-14T01:31:15.237 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.236+0000 7f23d422d640 -1 reset not still connected to 0x55af65e551e0 2023-10-14T01:31:15.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.236+0000 7f23d422d640 -1 reset not still connected to 0x55af66f26820 2023-10-14T01:31:15.238 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.236+0000 7f23d422d640 -1 reset not still connected to 0x55af66f27450 2023-10-14T01:31:15.266 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.264+0000 7f23d422d640 -1 reset not still connected to 0x55af85f80a90 2023-10-14T01:31:15.413 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.412+0000 7f23d422d640 -1 reset not still connected to 0x55af40c721a0 2023-10-14T01:31:15.451 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.448+0000 7f23d422d640 -1 reset not still connected to 0x55af4f6192b0 2023-10-14T01:31:15.455 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.456+0000 7f23d422d640 -1 reset not still connected to 0x55af5026a000 2023-10-14T01:31:15.456 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.456+0000 7f23d422d640 -1 reset not still connected to 0x55af5ef20270 2023-10-14T01:31:15.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.460+0000 7f23d422d640 -1 reset not still connected to 0x55af65e551e0 2023-10-14T01:31:15.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.460+0000 7f23d422d640 -1 reset not still connected to 0x55af66f26820 2023-10-14T01:31:15.462 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.460+0000 7f23d422d640 -1 reset not still connected to 0x55af66f27450 2023-10-14T01:31:15.483 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.476+0000 7f23d422d640 -1 reset not still connected to 0x55af7ca2c820 2023-10-14T01:31:15.484 INFO:tasks.ceph.osd.1.smithi138.stderr:2023-10-14T01:31:15.476+0000 7f23d422d640 -1 reset not still connected to 0x55af85f80a90 2023-10-14T01:31:16.040 INFO:tasks.ceph.osd.0:Stopped 2023-10-14T01:31:16.041 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-14T01:31:16.041 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:31:16.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-14T01:31:21.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-14T01:31:22.143 INFO:tasks.ceph.osd.1:Stopped 2023-10-14T01:31:22.144 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-14T01:31:22.144 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:31:26.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-14T01:31:26.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-14T01:31:28.246 INFO:tasks.ceph.osd.2:Stopped 2023-10-14T01:31:28.247 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-14T01:31:28.247 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-14T01:31:28.248 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:31:32.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-14T01:31:32.065 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-14T01:31:34.350 INFO:tasks.ceph.mgr.x:Stopped 2023-10-14T01:31:34.351 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-14T01:31:34.351 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-14T01:31:34.352 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:31:37.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-14T01:31:37.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-14T01:31:40.453 INFO:tasks.ceph.mon.a:Stopped 2023-10-14T01:31:40.454 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-14T01:31:40.454 DEBUG:teuthology.orchestra.run.smithi138:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-14T01:31:40.543 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi138.front.sepia.ceph.com 2023-10-14T01:31:40.543 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-14T01:31:41.153 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi138.front.sepia.ceph.com 2023-10-14T01:31:41.154 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-14T01:31:41.880 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi138.front.sepia.ceph.com 2023-10-14T01:31:41.881 DEBUG:teuthology.orchestra.run.smithi138:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-14T01:31:42.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-14T01:31:42.219 INFO:tasks.ceph:Archiving mon data... 2023-10-14T01:31:42.221 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257/data/mon.a.tgz 2023-10-14T01:31:42.221 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2023-10-14T01:31:42.240 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.dQdwD4QTF7 2023-10-14T01:31:42.241 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dQdwD4QTF7 2023-10-14T01:31:42.450 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.dQdwD4QTF7 2023-10-14T01:31:42.539 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.dQdwD4QTF7 is 683KB 2023-10-14T01:31:42.605 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.dQdwD4QTF7 2023-10-14T01:31:42.612 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-14T01:31:42.612 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-14T01:31:42.694 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-14T01:31:42.695 DEBUG:teuthology.orchestra.run.smithi138:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-14T01:31:42.755 INFO:teuthology.orchestra.run.smithi138.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-14T01:31:42.764 INFO:tasks.ceph:Archiving crash dumps... 2023-10-14T01:31:42.767 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257/remote/smithi138/crash 2023-10-14T01:31:42.768 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-14T01:31:42.817 INFO:tasks.ceph:Compressing logs... 2023-10-14T01:31:42.817 DEBUG:teuthology.orchestra.run.smithi138:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-14T01:31:42.894 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54659.log 2023-10-14T01:31:42.895 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42921.log 2023-10-14T01:31:42.895 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54659.log.gz 2023-10-14T01:31:42.895 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49797.log 2023-10-14T01:31:42.899 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53549.log 2023-10-14T01:31:42.899 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.49797.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38645.log 2023-10-14T01:31:42.900 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53549.log.gz 2023-10-14T01:31:42.900 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53738.log 2023-10-14T01:31:42.900 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53660.log 2023-10-14T01:31:42.900 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53738.log.gz 2023-10-14T01:31:42.901 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2023-10-14T01:31:42.901 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52669.log 2023-10-14T01:31:42.901 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53660.log.gz 2023-10-14T01:31:42.902 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43435.log 2023-10-14T01:31:42.902 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54259.log 2023-10-14T01:31:42.902 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52669.log.gz 2023-10-14T01:31:42.903 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52838.log 2023-10-14T01:31:42.903 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2023-10-14T01:31:42.903 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54259.log.gz 2023-10-14T01:31:42.904 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2023-10-14T01:31:42.904 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52838.log.gz 2023-10-14T01:31:42.904 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42972.log 2023-10-14T01:31:42.905 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49451.log 2023-10-14T01:31:42.905 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44652.log 2023-10-14T01:31:42.954 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39867.log 2023-10-14T01:31:42.955 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48853.log 2023-10-14T01:31:42.955 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45200.log 2023-10-14T01:31:42.955 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51102.log 2023-10-14T01:31:42.956 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41084.log 2023-10-14T01:31:42.956 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43524.log 2023-10-14T01:31:42.956 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54103.log 2023-10-14T01:31:42.956 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43462.log 2023-10-14T01:31:42.957 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44376.log 2023-10-14T01:31:42.957 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54103.log.gz 2023-10-14T01:31:42.957 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2023-10-14T01:31:42.958 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46139.log 2023-10-14T01:31:42.958 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47064.log 2023-10-14T01:31:42.958 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43435.log.gz 2023-10-14T01:31:42.959 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39867.log.gz -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2023-10-14T01:31:42.959 INFO:teuthology.orchestra.run.smithi138.stderr: 2023-10-14T01:31:42.959 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48853.log.gz 2023-10-14T01:31:42.959 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2023-10-14T01:31:42.960 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2023-10-14T01:31:42.960 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43524.log.gz 2023-10-14T01:31:42.960 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2023-10-14T01:31:42.961 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42921.log.gz 0.0% 2023-10-14T01:31:42.961 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41084.log.gz 2023-10-14T01:31:42.961 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43462.log.gz 2023-10-14T01:31:42.962 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44652.log.gz 2023-10-14T01:31:42.962 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51102.log.gz 2023-10-14T01:31:42.962 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46139.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49451.log.gz -- replaced with /var/log/ceph/ceph-client.admin.38645.log.gz 2023-10-14T01:31:42.963 INFO:teuthology.orchestra.run.smithi138.stderr: 2023-10-14T01:31:42.963 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45200.log.gz 2023-10-14T01:31:42.963 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56100.log 2023-10-14T01:31:42.963 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49797.log.gz 2023-10-14T01:31:42.964 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42972.log.gz 2023-10-14T01:31:42.964 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44376.log.gz 2023-10-14T01:31:42.965 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46139.log.gz 2023-10-14T01:31:42.965 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49825.log 2023-10-14T01:31:42.965 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47064.log.gz 2023-10-14T01:31:42.965 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50215.log 2023-10-14T01:31:42.966 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56100.log.gz 2023-10-14T01:31:42.966 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46968.log 2023-10-14T01:31:42.967 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49825.log.gz 2023-10-14T01:31:42.967 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43748.log 2023-10-14T01:31:42.967 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50215.log.gz 2023-10-14T01:31:42.967 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55805.log 2023-10-14T01:31:42.968 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46968.log.gz 2023-10-14T01:31:42.968 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48511.log 2023-10-14T01:31:42.968 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43748.log.gz 2023-10-14T01:31:42.969 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2023-10-14T01:31:42.969 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55805.log.gz 2023-10-14T01:31:42.969 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44087.log 2023-10-14T01:31:42.970 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48511.log.gz 2023-10-14T01:31:42.970 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50584.log 2023-10-14T01:31:42.970 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2023-10-14T01:31:42.971 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49298.log 2023-10-14T01:31:42.971 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44087.log.gz 2023-10-14T01:31:42.971 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50584.log.gz 2023-10-14T01:31:42.972 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52104.log 2023-10-14T01:31:42.972 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46848.log 2023-10-14T01:31:42.972 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49298.log.gz 2023-10-14T01:31:42.973 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38682.log 2023-10-14T01:31:42.973 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52104.log.gz 2023-10-14T01:31:42.973 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47279.log 2023-10-14T01:31:42.974 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46848.log.gz 2023-10-14T01:31:42.974 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53232.log 2023-10-14T01:31:42.974 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38682.log.gz 2023-10-14T01:31:42.974 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52226.log 2023-10-14T01:31:42.975 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47279.log.gz 2023-10-14T01:31:42.975 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.53232.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55061.log 2023-10-14T01:31:42.975 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53232.log.gz 2023-10-14T01:31:42.976 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52595.log 2023-10-14T01:31:42.976 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52226.log.gz 2023-10-14T01:31:42.976 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46908.log 2023-10-14T01:31:42.977 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55061.log.gz 2023-10-14T01:31:42.977 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42451.log 2023-10-14T01:31:42.977 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52595.log.gz 2023-10-14T01:31:42.978 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40119.log 2023-10-14T01:31:42.978 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46908.log.gz 2023-10-14T01:31:42.979 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2023-10-14T01:31:42.979 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42451.log.gz 2023-10-14T01:31:42.979 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49512.log 2023-10-14T01:31:42.979 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40119.log.gz 2023-10-14T01:31:42.980 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42697.log 2023-10-14T01:31:42.980 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2023-10-14T01:31:42.980 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42582.log 2023-10-14T01:31:42.981 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49512.log.gz 2023-10-14T01:31:42.981 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53124.log 2023-10-14T01:31:42.981 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42697.log.gz 2023-10-14T01:31:42.982 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37369.log 2023-10-14T01:31:42.982 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42582.log.gz 2023-10-14T01:31:42.982 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-14T01:31:42.983 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53124.log.gz 2023-10-14T01:31:42.983 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2023-10-14T01:31:42.983 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2023-10-14T01:31:42.983 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55534.log 2023-10-14T01:31:42.984 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51587.log 2023-10-14T01:31:42.984 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz 2023-10-14T01:31:42.984 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38574.log 2023-10-14T01:31:42.985 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55534.log.gz 2023-10-14T01:31:42.985 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44455.log 2023-10-14T01:31:42.985 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51587.log.gz 2023-10-14T01:31:42.985 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50520.log 2023-10-14T01:31:42.986 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38574.log.gz 2023-10-14T01:31:42.986 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44455.log.gz 2023-10-14T01:31:42.986 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50002.log 2023-10-14T01:31:42.987 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54953.log 2023-10-14T01:31:42.987 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50520.log.gz 2023-10-14T01:31:42.987 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45067.log 0.0% 2023-10-14T01:31:42.987 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50002.log.gz 2023-10-14T01:31:42.988 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54953.log.gz 2023-10-14T01:31:42.988 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52890.log 2023-10-14T01:31:42.988 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54207.log 2023-10-14T01:31:42.989 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45067.log.gz 2023-10-14T01:31:42.989 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49097.log 2023-10-14T01:31:42.989 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52890.log.gz 2023-10-14T01:31:42.990 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47181.log 2023-10-14T01:31:42.990 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54207.log.gz 2023-10-14T01:31:42.990 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54501.log 2023-10-14T01:31:42.991 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49097.log.gz 2023-10-14T01:31:42.991 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46633.log 2023-10-14T01:31:42.991 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47181.log.gz 2023-10-14T01:31:42.991 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50370.log 2023-10-14T01:31:42.992 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54501.log.gz 2023-10-14T01:31:42.992 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52864.log 2023-10-14T01:31:42.992 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46633.log.gz 2023-10-14T01:31:42.993 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47734.log 2023-10-14T01:31:42.993 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50370.log.gz 2023-10-14T01:31:42.993 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49342.log 2023-10-14T01:31:42.994 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52864.log.gz 2023-10-14T01:31:42.994 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52614.log 2023-10-14T01:31:42.994 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47734.log.gz 2023-10-14T01:31:42.995 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49342.log.gz 2023-10-14T01:31:42.995 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48686.log 2023-10-14T01:31:42.995 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40460.log 2023-10-14T01:31:42.995 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52614.log.gz 2023-10-14T01:31:42.996 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48686.log.gz 2023-10-14T01:31:42.996 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39205.log 2023-10-14T01:31:42.996 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51527.log 2023-10-14T01:31:42.997 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2023-10-14T01:31:42.997 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39168.log 2023-10-14T01:31:42.997 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39205.log.gz 2023-10-14T01:31:42.998 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45007.log 2023-10-14T01:31:42.998 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51527.log.gz 2023-10-14T01:31:42.998 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42041.log 2023-10-14T01:31:42.998 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39168.log.gz 2023-10-14T01:31:42.999 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2023-10-14T01:31:42.999 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45007.log.gz 2023-10-14T01:31:42.999 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43778.log 2023-10-14T01:31:43.000 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42041.log.gz 2023-10-14T01:31:43.000 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53868.log 2023-10-14T01:31:43.000 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2023-10-14T01:31:43.001 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38699.log 2023-10-14T01:31:43.001 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43778.log.gz 2023-10-14T01:31:43.002 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43411.log 2023-10-14T01:31:43.002 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53868.log.gz 2023-10-14T01:31:43.002 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38699.log.gz 2023-10-14T01:31:43.002 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56364.log 2023-10-14T01:31:43.003 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55992.log 2023-10-14T01:31:43.003 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43411.log.gz 2023-10-14T01:31:43.003 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56364.log.gz 2023-10-14T01:31:43.004 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55454.log 2023-10-14T01:31:43.004 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47251.log 2023-10-14T01:31:43.004 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55992.log.gz 2023-10-14T01:31:43.005 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55454.log.gz 2023-10-14T01:31:43.005 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38466.log 2023-10-14T01:31:43.005 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49023.log 2023-10-14T01:31:43.006 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47251.log.gz 2023-10-14T01:31:43.006 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52521.log 2023-10-14T01:31:43.006 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38466.log.gz 2023-10-14T01:31:43.007 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2023-10-14T01:31:43.007 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49023.log.gz 2023-10-14T01:31:43.007 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52916.log 2023-10-14T01:31:43.008 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52521.log.gz 2023-10-14T01:31:43.008 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48746.log 2023-10-14T01:31:43.008 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2023-10-14T01:31:43.008 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48905.log 2023-10-14T01:31:43.009 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52916.log.gz 2023-10-14T01:31:43.009 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45283.log 2023-10-14T01:31:43.009 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48746.log.gz 2023-10-14T01:31:43.010 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2023-10-14T01:31:43.010 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48905.log.gz 2023-10-14T01:31:43.010 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55375.log 2023-10-14T01:31:43.011 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45283.log.gz 2023-10-14T01:31:43.011 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55753.log 2023-10-14T01:31:43.011 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2023-10-14T01:31:43.012 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43718.log 2023-10-14T01:31:43.012 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55375.log.gz 2023-10-14T01:31:43.012 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2023-10-14T01:31:43.013 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55753.log.gz 2023-10-14T01:31:43.013 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48539.log 2023-10-14T01:31:43.013 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43718.log.gz 2023-10-14T01:31:43.013 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52777.log 2023-10-14T01:31:43.014 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2023-10-14T01:31:43.014 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42945.log 2023-10-14T01:31:43.014 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48539.log.gz 2023-10-14T01:31:43.015 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44159.log 2023-10-14T01:31:43.015 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52777.log.gz 2023-10-14T01:31:43.015 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48267.log 2023-10-14T01:31:43.016 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42945.log.gz 2023-10-14T01:31:43.016 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45648.log 2023-10-14T01:31:43.016 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44159.log.gz 2023-10-14T01:31:43.017 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38881.log 2023-10-14T01:31:43.017 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48267.log.gz 2023-10-14T01:31:43.017 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37748.log 2023-10-14T01:31:43.018 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45648.log.gz 2023-10-14T01:31:43.018 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48381.log 2023-10-14T01:31:43.018 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38881.log.gz 2023-10-14T01:31:43.019 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51918.log 2023-10-14T01:31:43.019 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37748.log.gz 2023-10-14T01:31:43.019 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49542.log 2023-10-14T01:31:43.020 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48381.log.gz 2023-10-14T01:31:43.020 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40750.log 2023-10-14T01:31:43.020 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51918.log.gz 2023-10-14T01:31:43.021 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54711.log 2023-10-14T01:31:43.021 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49542.log.gz 2023-10-14T01:31:43.021 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51858.log 2023-10-14T01:31:43.021 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gz 2023-10-14T01:31:43.022 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55006.log 2023-10-14T01:31:43.022 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54711.log.gz 2023-10-14T01:31:43.023 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52502.log 2023-10-14T01:31:43.023 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51858.log.gz 2023-10-14T01:31:43.023 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46447.log 2023-10-14T01:31:43.023 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55006.log.gz 2023-10-14T01:31:43.024 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53816.log 2023-10-14T01:31:43.024 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52502.log.gz 2023-10-14T01:31:43.025 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45802.log 2023-10-14T01:31:43.025 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46447.log.gz 2023-10-14T01:31:43.025 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52632.log 2023-10-14T01:31:43.025 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53816.log.gz 2023-10-14T01:31:43.026 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53973.log 2023-10-14T01:31:43.026 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45802.log.gz 2023-10-14T01:31:43.027 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46328.log 2023-10-14T01:31:43.027 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52632.log.gz 2023-10-14T01:31:43.027 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47151.log 2023-10-14T01:31:43.027 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53973.log.gz 2023-10-14T01:31:43.028 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55507.log 2023-10-14T01:31:43.028 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46328.log.gz 2023-10-14T01:31:43.028 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53258.log 2023-10-14T01:31:43.029 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47151.log.gz 2023-10-14T01:31:43.029 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54051.log 2023-10-14T01:31:43.029 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55507.log.gz 2023-10-14T01:31:43.030 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38446.log 2023-10-14T01:31:43.030 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53258.log.gz 2023-10-14T01:31:43.030 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54051.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-14T01:31:43.031 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54051.log.gz 2023-10-14T01:31:43.031 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39023.log 2023-10-14T01:31:43.031 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38446.log.gz 2023-10-14T01:31:43.032 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45097.log 2023-10-14T01:31:43.032 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48076.log 2023-10-14T01:31:43.032 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39023.log.gz 2023-10-14T01:31:43.033 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47806.log 2023-10-14T01:31:43.033 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45097.log.gz 2023-10-14T01:31:43.033 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34532.log 2023-10-14T01:31:43.033 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48076.log.gz 2023-10-14T01:31:43.034 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53206.log 2023-10-14T01:31:43.034 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47806.log.gz 2023-10-14T01:31:43.034 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.34532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44590.log 2023-10-14T01:31:43.035 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34532.log.gz 2023-10-14T01:31:43.035 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51888.log 2023-10-14T01:31:43.035 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53206.log.gz 2023-10-14T01:31:43.036 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53842.log 2023-10-14T01:31:43.036 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44590.log.gz 2023-10-14T01:31:43.036 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53493.log 2023-10-14T01:31:43.037 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51888.log.gz 2023-10-14T01:31:43.037 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47859.log 2023-10-14T01:31:43.037 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53842.log.gz 2023-10-14T01:31:43.038 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49726.log 2023-10-14T01:31:43.038 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53493.log.gz 2023-10-14T01:31:43.038 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56468.log 2023-10-14T01:31:43.038 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47859.log.gz 2023-10-14T01:31:43.039 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44241.log 2023-10-14T01:31:43.039 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49726.log.gz 2023-10-14T01:31:43.039 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48411.log 2023-10-14T01:31:43.040 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56468.log.gz 2023-10-14T01:31:43.040 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44735.log 2023-10-14T01:31:43.040 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44241.log.gz 2023-10-14T01:31:43.041 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37455.log 2023-10-14T01:31:43.041 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48411.log.gz 2023-10-14T01:31:43.041 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44823.log 2023-10-14T01:31:43.042 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44735.log.gz 2023-10-14T01:31:43.042 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5/var/log/ceph/ceph-client.admin.37455.log: --verbose -- /var/log/ceph/ceph-client.admin.34653.log 2023-10-14T01:31:43.042 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37455.log.gz 2023-10-14T01:31:43.042 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38827.log 2023-10-14T01:31:43.043 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34556.log 2023-10-14T01:31:43.043 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2023-10-14T01:31:43.043 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-14T01:31:43.044 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42339.log 2023-10-14T01:31:43.044 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48441.log 2023-10-14T01:31:43.044 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46821.log 2023-10-14T01:31:43.045 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39961.log 2023-10-14T01:31:43.045 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39741.log 2023-10-14T01:31:43.045 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41473.log 2023-10-14T01:31:43.045 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53764.log 2023-10-14T01:31:43.046 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55401.log 2023-10-14T01:31:43.046 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40087.log 2023-10-14T01:31:43.046 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37591.log 2023-10-14T01:31:43.047 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39541.log 2023-10-14T01:31:43.047 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45138.log 2023-10-14T01:31:43.047 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2023-10-14T01:31:43.048 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48634.log 2023-10-14T01:31:43.048 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49176.log 2023-10-14T01:31:43.048 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46276.log 2023-10-14T01:31:43.048 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56312.log 2023-10-14T01:31:43.049 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45744.log 2023-10-14T01:31:43.049 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39239.log 2023-10-14T01:31:43.049 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51557.log 2023-10-14T01:31:43.050 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46049.log 2023-10-14T01:31:43.050 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52705.log 2023-10-14T01:31:43.050 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2023-10-14T01:31:43.050 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53576.log 2023-10-14T01:31:43.051 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48962.log 2023-10-14T01:31:43.051 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55966.log 2023-10-14T01:31:43.051 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55590.log 2023-10-14T01:31:43.052 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52317.log 2023-10-14T01:31:43.052 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51041.log 2023-10-14T01:31:43.052 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51677.log 2023-10-14T01:31:43.053 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48808.log 2023-10-14T01:31:43.053 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41626.log 2023-10-14T01:31:43.053 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2023-10-14T01:31:43.053 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44793.log 2023-10-14T01:31:43.054 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53072.log 2023-10-14T01:31:43.054 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42483.log 2023-10-14T01:31:43.189 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53311.log 2023-10-14T01:31:43.189 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53632.log 2023-10-14T01:31:43.189 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43164.log 2023-10-14T01:31:43.190 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2023-10-14T01:31:43.190 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46796.log 2023-10-14T01:31:43.190 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54685.log 2023-10-14T01:31:43.190 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51223.log 2023-10-14T01:31:43.191 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51437.log 2023-10-14T01:31:43.191 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2023-10-14T01:31:43.191 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44762.log 2023-10-14T01:31:43.191 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53046.log 2023-10-14T01:31:43.192 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55113.log 2023-10-14T01:31:43.192 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43216.log 2023-10-14T01:31:43.192 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48992.log 2023-10-14T01:31:43.192 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56153.log 2023-10-14T01:31:43.193 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48658.log 2023-10-14T01:31:43.193 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2023-10-14T01:31:43.193 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2023-10-14T01:31:43.193 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55139.log 2023-10-14T01:31:43.194 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44823.log: /var/log/ceph/ceph-client.admin.34653.log: /var/log/ceph/ceph-client.admin.38827.log: /var/log/ceph/ceph-client.admin.34556.log: /var/log/ceph/ceph-client.admin.39131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34653.log.gz 0.0% 2023-10-14T01:31:43.194 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44823.log.gz 2023-10-14T01:31:43.194 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41332.log: /var/log/ceph/ceph-mon.a.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34556.log.gz 2023-10-14T01:31:43.194 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2023-10-14T01:31:43.195 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38827.log.gz 2023-10-14T01:31:43.195 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2023-10-14T01:31:43.195 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52448.log 2023-10-14T01:31:43.195 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42339.log: /var/log/ceph/ceph-client.admin.48441.log: /var/log/ceph/ceph-client.admin.46821.log: /var/log/ceph/ceph-client.admin.39961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42339.log.gz 2023-10-14T01:31:43.196 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48441.log.gz 2023-10-14T01:31:43.196 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39741.log: /var/log/ceph/ceph-client.admin.41473.log: /var/log/ceph/ceph-client.admin.53764.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39961.log.gz 2023-10-14T01:31:43.196 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46821.log.gz 2023-10-14T01:31:43.196 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41473.log.gz 2023-10-14T01:31:43.197 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39741.log.gz 2023-10-14T01:31:43.197 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51314.log 2023-10-14T01:31:43.197 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53764.log.gz 2023-10-14T01:31:43.197 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55401.log: /var/log/ceph/ceph-client.admin.55139.log: /var/log/ceph/ceph-client.admin.52448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55401.log.gz 2023-10-14T01:31:43.198 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.55139.log.gz 2023-10-14T01:31:43.198 INFO:teuthology.orchestra.run.smithi138.stderr: /var/log/ceph/ceph-client.admin.45588.log 2023-10-14T01:31:43.198 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52448.log.gz 2023-10-14T01:31:43.199 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40087.log: /var/log/ceph/ceph-client.admin.37591.log: /var/log/ceph/ceph-client.admin.39541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40087.log.gz 2023-10-14T01:31:43.199 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37591.log.gz 2023-10-14T01:31:43.199 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39541.log.gz 2023-10-14T01:31:43.199 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45138.log.gz 2023-10-14T01:31:43.200 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0%/var/log/ceph/ceph-client.admin.48634.log: -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2023-10-14T01:31:43.200 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51314.log.gz 2023-10-14T01:31:43.200 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45037.log/var/log/ceph/ceph-client.admin.56312.log: 2023-10-14T01:31:43.200 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48634.log.gz 2023-10-14T01:31:43.201 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49176.log.gz 2023-10-14T01:31:43.201 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46276.log.gz 2023-10-14T01:31:43.201 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56312.log.gz 2023-10-14T01:31:43.202 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45588.log: /var/log/ceph/ceph-client.admin.45744.log: /var/log/ceph/ceph-client.admin.39239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45588.log.gz 2023-10-14T01:31:43.202 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51557.log: /var/log/ceph/ceph-client.admin.46049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50185.log 2023-10-14T01:31:43.202 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45744.log.gz 2023-10-14T01:31:43.202 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39239.log.gz 2023-10-14T01:31:43.203 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51557.log.gz 2023-10-14T01:31:43.203 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46049.log.gz -- replaced with /var/log/ceph/ceph-client.admin.52705.log.gz 2023-10-14T01:31:43.203 INFO:teuthology.orchestra.run.smithi138.stderr: 2023-10-14T01:31:43.203 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2023-10-14T01:31:43.204 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.53576.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46723.log 2023-10-14T01:31:43.204 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48962.log: /var/log/ceph/ceph-client.admin.55966.log: /var/log/ceph/ceph-client.admin.45037.log: /var/log/ceph/ceph-client.admin.55590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53576.log.gz 2023-10-14T01:31:43.204 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48962.log.gz 2023-10-14T01:31:43.205 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55966.log.gz 2023-10-14T01:31:43.205 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45037.log.gz 2023-10-14T01:31:43.205 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55590.log.gz 2023-10-14T01:31:43.205 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50185.log: /var/log/ceph/ceph-client.admin.52317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45501.log 2023-10-14T01:31:43.206 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51041.log: /var/log/ceph/ceph-client.admin.51677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50185.log.gz 2023-10-14T01:31:43.206 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52317.log.gz 0.0% 2023-10-14T01:31:43.206 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51041.log.gz 2023-10-14T01:31:43.206 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51677.log.gz 2023-10-14T01:31:43.207 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41626.log: /var/log/ceph/ceph-client.admin.41656.log: /var/log/ceph/ceph-client.admin.44793.log: /var/log/ceph/ceph-client.admin.46723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41626.log.gz 2023-10-14T01:31:43.207 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53072.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz --verbose 2023-10-14T01:31:43.207 INFO:teuthology.orchestra.run.smithi138.stderr: -- 0.0% /var/log/ceph/ceph-client.admin.42121.log -- replaced with /var/log/ceph/ceph-client.admin.44793.log.gz 2023-10-14T01:31:43.208 INFO:teuthology.orchestra.run.smithi138.stderr: 2023-10-14T01:31:43.208 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46723.log.gz 2023-10-14T01:31:43.208 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53072.log.gz 2023-10-14T01:31:43.208 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42483.log.gz 2023-10-14T01:31:43.209 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48808.log.gz 2023-10-14T01:31:43.209 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53999.log 2023-10-14T01:31:43.209 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53311.log: /var/log/ceph/ceph-client.admin.53632.log: /var/log/ceph/ceph-client.admin.43164.log: /var/log/ceph/ceph-client.admin.38591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45501.log.gz 2023-10-14T01:31:43.210 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53311.log.gz 2023-10-14T01:31:43.210 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53632.log.gz 2023-10-14T01:31:43.210 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gz 2023-10-14T01:31:43.210 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2023-10-14T01:31:43.211 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42121.log: /var/log/ceph/ceph-client.admin.46796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42121.log.gz 2023-10-14T01:31:43.211 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54685.log: /var/log/ceph/ceph-client.admin.51223.log: /var/log/ceph/ceph-client.admin.51437.log: /var/log/ceph/ceph-client.admin.40706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46796.log.gz 2023-10-14T01:31:43.211 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51223.log.gz 2023-10-14T01:31:43.212 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54685.log.gz 2023-10-14T01:31:43.212 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41012.log 2023-10-14T01:31:43.212 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51437.log.gz 2023-10-14T01:31:43.212 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2023-10-14T01:31:43.213 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44762.log: /var/log/ceph/ceph-client.admin.53046.log: /var/log/ceph/ceph-client.admin.55113.log: /var/log/ceph/ceph-client.admin.43216.log: /var/log/ceph/ceph-client.admin.48992.log: /var/log/ceph/ceph-client.admin.53999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44762.log.gz 2023-10-14T01:31:43.213 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53046.log.gz 2023-10-14T01:31:43.213 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55113.log.gz 2023-10-14T01:31:43.213 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2023-10-14T01:31:43.214 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43216.log.gz 2023-10-14T01:31:43.214 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48992.log.gz 2023-10-14T01:31:43.214 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0%/var/log/ceph/ceph-client.admin.56153.log: /var/log/ceph/ceph-client.admin.48658.log: /var/log/ceph/ceph-client.admin.40379.log: -- replaced with /var/log/ceph/ceph-client.admin.53999.log.gz 2023-10-14T01:31:43.215 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56153.log.gz 2023-10-14T01:31:43.215 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48658.log.gz 2023-10-14T01:31:43.215 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz 2023-10-14T01:31:43.215 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44853.log 2023-10-14T01:31:43.216 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41012.log.gz 2023-10-14T01:31:43.216 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52012.log 2023-10-14T01:31:43.216 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2023-10-14T01:31:43.216 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46109.log 2023-10-14T01:31:43.217 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44853.log.gz 2023-10-14T01:31:43.217 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52012.log.gz 2023-10-14T01:31:43.217 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47334.log 2023-10-14T01:31:43.217 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46179.log 2023-10-14T01:31:43.218 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46109.log.gz 2023-10-14T01:31:43.218 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47334.log: -- /var/log/ceph/ceph-client.admin.43136.log 2023-10-14T01:31:43.218 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47334.log.gz 2023-10-14T01:31:43.218 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.46179.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51375.log 2023-10-14T01:31:43.219 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46179.log.gz 2023-10-14T01:31:43.219 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48103.log 2023-10-14T01:31:43.219 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43136.log.gz 2023-10-14T01:31:43.219 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47672.log 2023-10-14T01:31:43.220 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51375.log.gz 2023-10-14T01:31:43.220 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44332.log 2023-10-14T01:31:43.220 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48103.log.gz 2023-10-14T01:31:43.220 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44549.log 2023-10-14T01:31:43.221 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47672.log.gz 2023-10-14T01:31:43.221 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52759.log 2023-10-14T01:31:43.221 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44332.log.gz 2023-10-14T01:31:43.221 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45259.log 2023-10-14T01:31:43.222 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44549.log.gz 2023-10-14T01:31:43.222 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43838.log 2023-10-14T01:31:43.222 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52759.log.gz 2023-10-14T01:31:43.222 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47496.log 2023-10-14T01:31:43.223 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45259.log.gz 2023-10-14T01:31:43.223 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54313.log 2023-10-14T01:31:43.223 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43838.log.gz 2023-10-14T01:31:43.224 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55619.log 2023-10-14T01:31:43.224 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47496.log.gz 2023-10-14T01:31:43.224 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43910.log 2023-10-14T01:31:43.224 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54313.log.gz 2023-10-14T01:31:43.225 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.55619.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42391.log 2023-10-14T01:31:43.225 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55619.log.gz 2023-10-14T01:31:43.225 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48024.log 2023-10-14T01:31:43.225 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43910.log.gz 2023-10-14T01:31:43.226 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55779.log 2023-10-14T01:31:43.226 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42391.log.gz 2023-10-14T01:31:43.226 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56018.log 2023-10-14T01:31:43.226 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48024.log.gz 2023-10-14T01:31:43.227 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2023-10-14T01:31:43.227 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55779.log.gz 2023-10-14T01:31:43.227 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49890.log 2023-10-14T01:31:43.227 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56018.log.gz 2023-10-14T01:31:43.228 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43691.log 2023-10-14T01:31:43.228 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2023-10-14T01:31:43.228 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44301.log 2023-10-14T01:31:43.228 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49890.log.gz 2023-10-14T01:31:43.229 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43691.log: /var/log/ceph/ceph-client.admin.52196.log 2023-10-14T01:31:43.229 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43691.log.gz 2023-10-14T01:31:43.229 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55673.log 2023-10-14T01:31:43.229 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44301.log.gz 2023-10-14T01:31:43.230 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48471.log 2023-10-14T01:31:43.230 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52196.log.gz 2023-10-14T01:31:43.230 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39330.log 2023-10-14T01:31:43.231 INFO:teuthology.orchestra.run.smithi138.stderr: 27.9% -- replaced with /var/log/ceph/ceph-client.admin.55673.log.gz 2023-10-14T01:31:43.231 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50980.log 2023-10-14T01:31:43.231 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz 2023-10-14T01:31:43.231 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49370.log 2023-10-14T01:31:43.232 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39330.log.gz 2023-10-14T01:31:43.232 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54233.log 2023-10-14T01:31:43.232 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50980.log.gz 2023-10-14T01:31:43.232 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51737.log 2023-10-14T01:31:43.233 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49370.log.gz 2023-10-14T01:31:43.233 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51828.log 2023-10-14T01:31:43.233 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54233.log.gz 2023-10-14T01:31:43.233 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56444.log 2023-10-14T01:31:43.234 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51737.log.gz 2023-10-14T01:31:43.234 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55857.log 2023-10-14T01:31:43.234 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51828.log.gz 2023-10-14T01:31:43.234 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54980.log 2023-10-14T01:31:43.235 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56444.log.gz 2023-10-14T01:31:43.235 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45893.log 2023-10-14T01:31:43.235 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55857.log.gz 2023-10-14T01:31:43.236 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51497.log 2023-10-14T01:31:43.236 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54980.log.gz 2023-10-14T01:31:43.236 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50093.log 2023-10-14T01:31:43.236 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45893.log.gz 2023-10-14T01:31:43.236 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51497.log.gz 2023-10-14T01:31:43.237 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46693.log 2023-10-14T01:31:43.237 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42759.log 2023-10-14T01:31:43.237 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50093.log.gz 2023-10-14T01:31:43.237 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42421.log 2023-10-14T01:31:43.238 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46693.log.gz 2023-10-14T01:31:43.238 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54129.log 2023-10-14T01:31:43.238 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42759.log.gz 2023-10-14T01:31:43.239 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37871.log 2023-10-14T01:31:43.239 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42421.log.gz 2023-10-14T01:31:43.239 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47091.log 2023-10-14T01:31:43.239 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54129.log.gz 2023-10-14T01:31:43.240 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2023-10-14T01:31:43.240 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37871.log.gz 2023-10-14T01:31:43.240 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55271.log 2023-10-14T01:31:43.240 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47091.log.gz 2023-10-14T01:31:43.241 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51193.log 2023-10-14T01:31:43.241 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2023-10-14T01:31:43.241 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53389.log 2023-10-14T01:31:43.241 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55271.log.gz 2023-10-14T01:31:43.242 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38807.log 2023-10-14T01:31:43.242 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51193.log.gz 2023-10-14T01:31:43.242 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39573.log 2023-10-14T01:31:43.242 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53389.log.gz 2023-10-14T01:31:43.243 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38807.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.38807.log.gz --verbose 2023-10-14T01:31:43.243 INFO:teuthology.orchestra.run.smithi138.stderr: -- /var/log/ceph/ceph-client.admin.55562.log 2023-10-14T01:31:43.243 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45342.log 2023-10-14T01:31:43.243 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39573.log.gz 2023-10-14T01:31:43.244 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2023-10-14T01:31:43.244 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55562.log.gz 2023-10-14T01:31:43.244 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45342.log.gz 2023-10-14T01:31:43.244 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43639.log 2023-10-14T01:31:43.245 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54898.log 2023-10-14T01:31:43.245 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2023-10-14T01:31:43.245 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38426.log 2023-10-14T01:31:43.245 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43639.log.gz 2023-10-14T01:31:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48194.log 2023-10-14T01:31:43.246 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54898.log.gz 2023-10-14T01:31:43.246 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54025.log 2023-10-14T01:31:43.246 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38426.log.gz 2023-10-14T01:31:43.247 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38861.log 2023-10-14T01:31:43.247 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48194.log.gz 2023-10-14T01:31:43.247 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54025.log.gz 2023-10-14T01:31:43.247 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43338.log 2023-10-14T01:31:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38861.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-14T01:31:43.248 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38861.log.gz 2023-10-14T01:31:43.248 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55193.log 2023-10-14T01:31:43.248 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43338.log.gz 2023-10-14T01:31:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50063.log 2023-10-14T01:31:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46603.log 2023-10-14T01:31:43.249 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55193.log.gz 2023-10-14T01:31:43.249 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40583.log 2023-10-14T01:31:43.250 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50063.log.gz 2023-10-14T01:31:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55349.log 2023-10-14T01:31:43.250 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46603.log.gz 2023-10-14T01:31:43.250 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50826.log 2023-10-14T01:31:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40583.log.gz 2023-10-14T01:31:43.251 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56338.log 2023-10-14T01:31:43.251 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55349.log.gz 2023-10-14T01:31:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44056.log 2023-10-14T01:31:43.252 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50826.log.gz 2023-10-14T01:31:43.252 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41108.log 2023-10-14T01:31:43.252 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56338.log.gz 2023-10-14T01:31:43.253 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44618.log 2023-10-14T01:31:43.253 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44056.log.gz 2023-10-14T01:31:43.253 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40982.log 2023-10-14T01:31:43.254 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41108.log.gz 2023-10-14T01:31:43.254 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50033.log/var/log/ceph/ceph-client.admin.44618.log: 2023-10-14T01:31:43.254 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44618.log.gz 2023-10-14T01:31:43.254 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.40982.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54872.log 2023-10-14T01:31:43.255 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40982.log.gz 2023-10-14T01:31:43.255 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50033.log.gz 2023-10-14T01:31:43.255 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49125.log 2023-10-14T01:31:43.255 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49859.log 2023-10-14T01:31:43.256 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54872.log.gz 2023-10-14T01:31:43.256 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51405.log 2023-10-14T01:31:43.256 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49125.log.gz 2023-10-14T01:31:43.256 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49859.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.49859.log.gz /var/log/ceph/ceph-client.admin.54155.log 2023-10-14T01:31:43.257 INFO:teuthology.orchestra.run.smithi138.stderr: 2023-10-14T01:31:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39060.log 2023-10-14T01:31:43.257 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51405.log.gz 2023-10-14T01:31:43.257 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2023-10-14T01:31:43.258 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54155.log.gz 2023-10-14T01:31:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37819.log 2023-10-14T01:31:43.258 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39060.log.gz 2023-10-14T01:31:43.258 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48164.log 2023-10-14T01:31:43.259 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2023-10-14T01:31:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40351.log 2023-10-14T01:31:43.259 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37819.log.gz 2023-10-14T01:31:43.259 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52539.log 2023-10-14T01:31:43.260 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48164.log.gz 2023-10-14T01:31:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2023-10-14T01:31:43.260 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40351.log.gz 2023-10-14T01:31:43.260 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50125.log 2023-10-14T01:31:43.261 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52539.log.gz 2023-10-14T01:31:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54475.log 2023-10-14T01:31:43.261 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2023-10-14T01:31:43.261 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45863.log 2023-10-14T01:31:43.262 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50125.log.gz 2023-10-14T01:31:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41229.log 2023-10-14T01:31:43.262 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54475.log.gz 2023-10-14T01:31:43.262 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52043.log 2023-10-14T01:31:43.263 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45863.log.gz 2023-10-14T01:31:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51071.log 2023-10-14T01:31:43.263 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41229.log.gz 2023-10-14T01:31:43.263 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52043.log.gz 2023-10-14T01:31:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54633.log 2023-10-14T01:31:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42069.log 2023-10-14T01:31:43.264 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51071.log.gz 2023-10-14T01:31:43.264 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55428.log 2023-10-14T01:31:43.264 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54633.log.gz 2023-10-14T01:31:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42069.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30856.log 2023-10-14T01:31:43.265 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42069.log.gz 2023-10-14T01:31:43.265 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37430.log 2023-10-14T01:31:43.265 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55428.log.gz 2023-10-14T01:31:43.266 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.tmp-client.admin.30856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42861.log 2023-10-14T01:31:43.266 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30856.log.gz 2023-10-14T01:31:43.266 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2023-10-14T01:31:43.266 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37430.log.gz 2023-10-14T01:31:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41902.log 2023-10-14T01:31:43.267 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42861.log.gz 2023-10-14T01:31:43.267 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52375.log 2023-10-14T01:31:43.267 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2023-10-14T01:31:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49238.log 2023-10-14T01:31:43.268 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41902.log.gz 2023-10-14T01:31:43.268 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39783.log 2023-10-14T01:31:43.268 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52375.log.gz 2023-10-14T01:31:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50675.log 2023-10-14T01:31:43.269 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49238.log.gz 2023-10-14T01:31:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39438.log 2023-10-14T01:31:43.269 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39783.log.gz 2023-10-14T01:31:43.269 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42827.log 2023-10-14T01:31:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50675.log.gz 2023-10-14T01:31:43.270 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42237.log 2023-10-14T01:31:43.270 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39438.log.gz 2023-10-14T01:31:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46478.log 2023-10-14T01:31:43.271 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42827.log.gz 2023-10-14T01:31:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49638.log 2023-10-14T01:31:43.271 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42237.log.gz 2023-10-14T01:31:43.272 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38753.log 2023-10-14T01:31:43.272 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46478.log.gz 2023-10-14T01:31:43.272 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52815.log 2023-10-14T01:31:43.272 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49638.log.gz 2023-10-14T01:31:43.273 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55699.log 2023-10-14T01:31:43.273 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38753.log.gz 2023-10-14T01:31:43.273 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38952.log 2023-10-14T01:31:43.273 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52815.log.gz 2023-10-14T01:31:43.274 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56391.log 2023-10-14T01:31:43.274 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55699.log.gz 2023-10-14T01:31:43.274 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40643.log 2023-10-14T01:31:43.275 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38952.log.gz 2023-10-14T01:31:43.275 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44489.log 2023-10-14T01:31:43.275 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56391.log.gz 2023-10-14T01:31:43.275 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40643.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40643.log.gz 2023-10-14T01:31:43.276 INFO:teuthology.orchestra.run.smithi138.stderr: /var/log/ceph/ceph-mgr.x.log 2023-10-14T01:31:43.276 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52256.log 2023-10-14T01:31:43.276 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44489.log.gz 2023-10-14T01:31:43.277 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47457.log 2023-10-14T01:31:43.277 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39825.log 2023-10-14T01:31:43.277 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52256.log.gz 2023-10-14T01:31:43.277 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41383.log 2023-10-14T01:31:43.278 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47457.log.gz 2023-10-14T01:31:43.278 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52430.log 2023-10-14T01:31:43.278 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39825.log.gz 2023-10-14T01:31:43.278 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2023-10-14T01:31:43.279 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41383.log.gz 2023-10-14T01:31:43.279 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48134.log 2023-10-14T01:31:43.279 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52430.log.gz 2023-10-14T01:31:43.279 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38520.log 2023-10-14T01:31:43.280 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz 2023-10-14T01:31:43.280 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51647.log 2023-10-14T01:31:43.280 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48134.log.gz 2023-10-14T01:31:43.280 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44428.log 2023-10-14T01:31:43.281 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38520.log.gz 2023-10-14T01:31:43.281 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47427.log 2023-10-14T01:31:43.281 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51647.log.gz 2023-10-14T01:31:43.281 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54527.log 2023-10-14T01:31:43.282 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54789.log 2023-10-14T01:31:43.282 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47427.log.gz 2023-10-14T01:31:43.282 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37663.log 2023-10-14T01:31:43.283 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54527.log.gz 2023-10-14T01:31:43.283 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2023-10-14T01:31:43.283 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54789.log.gz 2023-10-14T01:31:43.284 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40430.log 2023-10-14T01:31:43.284 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37663.log.gz 2023-10-14T01:31:43.284 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49207.log 2023-10-14T01:31:43.284 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2023-10-14T01:31:43.285 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53285.log 2023-10-14T01:31:43.285 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40430.log.gz 2023-10-14T01:31:43.285 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-14T01:31:43.285 INFO:teuthology.orchestra.run.smithi138.stderr: 26.5% -- replaced with /var/log/ceph/ceph-client.admin.44428.log.gz 2023-10-14T01:31:43.286 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49207.log.gz 2023-10-14T01:31:43.286 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43878.log 2023-10-14T01:31:43.286 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53285.log.gz 2023-10-14T01:31:43.286 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42610.log 2023-10-14T01:31:43.287 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44683.log 2023-10-14T01:31:43.287 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43878.log.gz 2023-10-14T01:31:43.287 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42610.log.gz 2023-10-14T01:31:43.288 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56286.log 2023-10-14T01:31:43.288 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44683.log.gz 2023-10-14T01:31:43.288 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2023-10-14T01:31:43.288 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5/var/log/ceph/ceph-client.admin.56286.log: --verbose -- /var/log/ceph/ceph-client.admin.45692.log 2023-10-14T01:31:43.289 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56286.log.gz 2023-10-14T01:31:43.289 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2023-10-14T01:31:43.289 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2023-10-14T01:31:43.289 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48224.log 2023-10-14T01:31:43.290 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45692.log.gz 2023-10-14T01:31:43.290 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49421.log 2023-10-14T01:31:43.290 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2023-10-14T01:31:43.290 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47584.log 2023-10-14T01:31:43.291 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48224.log.gz 2023-10-14T01:31:43.291 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42789.log 2023-10-14T01:31:43.291 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49421.log.gz 2023-10-14T01:31:43.291 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41748.log 2023-10-14T01:31:43.292 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47584.log.gz 2023-10-14T01:31:43.292 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52393.log 2023-10-14T01:31:43.292 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42789.log.gz 2023-10-14T01:31:43.292 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37533.log 2023-10-14T01:31:43.293 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41748.log.gz 2023-10-14T01:31:43.293 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56179.log 2023-10-14T01:31:43.293 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52393.log.gz 2023-10-14T01:31:43.293 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39222.log 2023-10-14T01:31:43.294 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37533.log.gz 2023-10-14T01:31:43.294 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2023-10-14T01:31:43.294 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56179.log.gz 2023-10-14T01:31:43.294 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2023-10-14T01:31:43.295 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39222.log.gz 2023-10-14T01:31:43.295 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50307.log 2023-10-14T01:31:43.295 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2023-10-14T01:31:43.295 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51767.log 2023-10-14T01:31:43.296 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2023-10-14T01:31:43.296 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42727.log 2023-10-14T01:31:43.296 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50307.log.gz 2023-10-14T01:31:43.296 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37741.log 2023-10-14T01:31:43.297 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51767.log.gz 2023-10-14T01:31:43.297 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41443.log 2023-10-14T01:31:43.297 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42727.log.gz 2023-10-14T01:31:43.297 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49696.log 2023-10-14T01:31:43.298 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37741.log.gz 2023-10-14T01:31:43.298 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2023-10-14T01:31:43.298 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2023-10-14T01:31:43.298 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40889.log 2023-10-14T01:31:43.299 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49696.log.gz 2023-10-14T01:31:43.299 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38392.log 2023-10-14T01:31:43.299 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2023-10-14T01:31:43.299 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2023-10-14T01:31:43.300 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz 2023-10-14T01:31:43.300 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49969.log 2023-10-14T01:31:43.300 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38392.log.gz 2023-10-14T01:31:43.300 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54077.log 2023-10-14T01:31:43.301 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2023-10-14T01:31:43.301 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52074.log 2023-10-14T01:31:43.301 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49969.log.gz 2023-10-14T01:31:43.302 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.54077.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50948.log 2023-10-14T01:31:43.302 INFO:teuthology.orchestra.run.smithi138.stderr: 89.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-14T01:31:43.302 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54077.log.gz 2023-10-14T01:31:43.302 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2023-10-14T01:31:43.303 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52074.log.gz 2023-10-14T01:31:43.303 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50430.log 2023-10-14T01:31:43.303 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50948.log.gz 2023-10-14T01:31:43.303 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54391.log 2023-10-14T01:31:43.303 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gz 2023-10-14T01:31:43.304 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43062.log 2023-10-14T01:31:43.304 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50430.log.gz 2023-10-14T01:31:43.304 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54391.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.54391.log.gz --verbose 2023-10-14T01:31:43.304 INFO:teuthology.orchestra.run.smithi138.stderr: -- /var/log/ceph/ceph-client.admin.41575.log 2023-10-14T01:31:43.305 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51284.log 2023-10-14T01:31:43.305 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43062.log.gz 2023-10-14T01:31:43.305 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43382.log 2023-10-14T01:31:43.305 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2023-10-14T01:31:43.306 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51467.log 2023-10-14T01:31:43.306 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51284.log.gz 2023-10-14T01:31:43.306 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53098.log 2023-10-14T01:31:43.306 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43382.log.gz 2023-10-14T01:31:43.307 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41543.log 2023-10-14T01:31:43.307 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51467.log.gz 2023-10-14T01:31:43.307 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53150.log 2023-10-14T01:31:43.307 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53098.log.gz 2023-10-14T01:31:43.308 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48932.log 2023-10-14T01:31:43.308 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41543.log.gz 2023-10-14T01:31:43.308 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2023-10-14T01:31:43.309 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53150.log.gz 2023-10-14T01:31:43.309 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53520.log 2023-10-14T01:31:43.309 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48932.log.gz 2023-10-14T01:31:43.309 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51344.log 2023-10-14T01:31:43.310 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2023-10-14T01:31:43.310 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50645.log 2023-10-14T01:31:43.310 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53520.log.gz 2023-10-14T01:31:43.310 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46386.log 2023-10-14T01:31:43.311 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51344.log.gz 2023-10-14T01:31:43.311 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50645.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50645.log.gz 2023-10-14T01:31:43.311 INFO:teuthology.orchestra.run.smithi138.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38736.log 2023-10-14T01:31:43.311 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2023-10-14T01:31:43.312 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46386.log.gz 2023-10-14T01:31:43.312 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53467.log 2023-10-14T01:31:43.312 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38736.log.gz 2023-10-14T01:31:43.312 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38611.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz -5 2023-10-14T01:31:43.313 INFO:teuthology.orchestra.run.smithi138.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41792.log 2023-10-14T01:31:43.313 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39605.log 2023-10-14T01:31:43.313 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53467.log.gz 2023-10-14T01:31:43.313 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52741.log 2023-10-14T01:31:43.314 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41792.log.gz 2023-10-14T01:31:43.314 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44183.log 2023-10-14T01:31:43.314 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39605.log.gz 2023-10-14T01:31:43.315 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42267.log 2023-10-14T01:31:43.315 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52741.log.gz 2023-10-14T01:31:43.315 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49268.log 2023-10-14T01:31:43.315 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44183.log.gz 2023-10-14T01:31:43.316 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42267.log.gz 2023-10-14T01:31:43.316 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51707.log 2023-10-14T01:31:43.316 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46304.log 2023-10-14T01:31:43.316 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49268.log.gz 2023-10-14T01:31:43.317 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2023-10-14T01:31:43.317 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51707.log.gz 2023-10-14T01:31:43.317 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2023-10-14T01:31:43.318 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46304.log.gz 2023-10-14T01:31:43.318 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44404.log 2023-10-14T01:31:43.318 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2023-10-14T01:31:43.318 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46551.log 2023-10-14T01:31:43.319 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2023-10-14T01:31:43.319 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54365.log 2023-10-14T01:31:43.319 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44404.log.gz 2023-10-14T01:31:43.320 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46417.log 2023-10-14T01:31:43.320 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46551.log.gz 2023-10-14T01:31:43.320 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54365.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54365.log.gz -5 2023-10-14T01:31:43.320 INFO:teuthology.orchestra.run.smithi138.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53441.log 2023-10-14T01:31:43.321 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49614.log 2023-10-14T01:31:43.321 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46417.log.gz 2023-10-14T01:31:43.321 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55647.log 2023-10-14T01:31:43.322 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53441.log.gz 2023-10-14T01:31:43.322 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49614.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.37559.log -- replaced with /var/log/ceph/ceph-client.admin.49614.log.gz 2023-10-14T01:31:43.322 INFO:teuthology.orchestra.run.smithi138.stderr: 2023-10-14T01:31:43.322 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48323.log 2023-10-14T01:31:43.323 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55647.log.gz 2023-10-14T01:31:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48350.log 2023-10-14T01:31:43.323 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37559.log.gz 2023-10-14T01:31:43.323 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48323.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48323.log.gz -5 2023-10-14T01:31:43.324 INFO:teuthology.orchestra.run.smithi138.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43554.log 2023-10-14T01:31:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50916.log 2023-10-14T01:31:43.324 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48350.log.gz 2023-10-14T01:31:43.324 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43554.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.43667.log 2023-10-14T01:31:43.325 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43554.log.gz 2023-10-14T01:31:43.325 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43584.log 2023-10-14T01:31:43.325 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42513.log 2023-10-14T01:31:43.325 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43667.log.gz 2023-10-14T01:31:43.326 INFO:teuthology.orchestra.run.smithi138.stderr: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.50916.log.gz 2023-10-14T01:31:43.326 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2023-10-14T01:31:43.326 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43584.log.gz 2023-10-14T01:31:43.326 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40297.log 2023-10-14T01:31:43.327 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42513.log.gz 2023-10-14T01:31:43.327 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39455.log 2023-10-14T01:31:43.327 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2023-10-14T01:31:43.327 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54181.log 2023-10-14T01:31:43.328 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40297.log.gz 2023-10-14T01:31:43.328 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54554.log 2023-10-14T01:31:43.328 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39455.log.gz 2023-10-14T01:31:43.328 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54181.log.gz 2023-10-14T01:31:43.329 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2023-10-14T01:31:43.329 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52557.log 2023-10-14T01:31:43.329 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54554.log.gz 2023-10-14T01:31:43.329 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54737.log 2023-10-14T01:31:43.330 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2023-10-14T01:31:43.330 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43308.log 2023-10-14T01:31:43.330 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52557.log.gz 2023-10-14T01:31:43.330 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49053.log 2023-10-14T01:31:43.331 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54737.log.gz 2023-10-14T01:31:43.331 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43308.log.gz 2023-10-14T01:31:43.331 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45166.log 2023-10-14T01:31:43.332 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55323.log 2023-10-14T01:31:43.332 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49053.log.gz 2023-10-14T01:31:43.332 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43277.log 2023-10-14T01:31:43.332 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45166.log.gz 2023-10-14T01:31:43.333 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52723.log 2023-10-14T01:31:43.333 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55323.log.gz 2023-10-14T01:31:43.333 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39421.log 2023-10-14T01:31:43.333 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43277.log.gz 2023-10-14T01:31:43.334 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51951.log 2023-10-14T01:31:43.334 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52723.log.gz 2023-10-14T01:31:43.334 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40403.log 2023-10-14T01:31:43.334 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39421.log.gz 2023-10-14T01:31:43.335 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52412.log 2023-10-14T01:31:43.335 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51951.log.gz 2023-10-14T01:31:43.335 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50735.log 2023-10-14T01:31:43.336 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40403.log.gz 2023-10-14T01:31:43.336 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2023-10-14T01:31:43.336 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52412.log.gz 2023-10-14T01:31:43.336 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47950.log 2023-10-14T01:31:43.337 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50735.log.gz 2023-10-14T01:31:43.337 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.38557.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42151.log 2023-10-14T01:31:43.337 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2023-10-14T01:31:43.337 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50276.log 2023-10-14T01:31:43.338 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47950.log.gz 2023-10-14T01:31:43.338 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46938.log 2023-10-14T01:31:43.338 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42151.log.gz 2023-10-14T01:31:43.338 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2023-10-14T01:31:43.339 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50276.log.gz 2023-10-14T01:31:43.339 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45231.log 2023-10-14T01:31:43.339 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46938.log.gz 2023-10-14T01:31:43.339 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46207.log 2023-10-14T01:31:43.340 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2023-10-14T01:31:43.340 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52797.log/var/log/ceph/ceph-client.admin.45231.log: 2023-10-14T01:31:43.340 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45231.log.gz 2023-10-14T01:31:43.340 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46207.log.gz 2023-10-14T01:31:43.341 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39493.log 2023-10-14T01:31:43.341 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44131.log 2023-10-14T01:31:43.341 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52797.log.gz 2023-10-14T01:31:43.342 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55481.log 2023-10-14T01:31:43.342 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39493.log.gz 2023-10-14T01:31:43.342 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39043.log 2023-10-14T01:31:43.342 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44131.log.gz 2023-10-14T01:31:43.343 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55727.log 2023-10-14T01:31:43.343 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55481.log.gz 2023-10-14T01:31:43.343 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39043.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-14T01:31:43.343 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39043.log.gz 2023-10-14T01:31:43.344 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53894.log 2023-10-14T01:31:43.344 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55727.log.gz 2023-10-14T01:31:43.344 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54845.log 2023-10-14T01:31:43.345 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.53894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50856.log 2023-10-14T01:31:43.345 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53894.log.gz 2023-10-14T01:31:43.345 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47704.log 2023-10-14T01:31:43.345 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54845.log.gz 2023-10-14T01:31:43.346 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50856.log: -- /var/log/ceph/ceph-client.admin.53712.log 2023-10-14T01:31:43.346 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50856.log.gz 2023-10-14T01:31:43.346 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2023-10-14T01:31:43.346 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47704.log.gz 2023-10-14T01:31:43.347 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50400.log 2023-10-14T01:31:43.347 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53712.log.gz 2023-10-14T01:31:43.347 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2023-10-14T01:31:43.347 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56260.log 2023-10-14T01:31:43.348 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56072.log 2023-10-14T01:31:43.348 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50400.log.gz 2023-10-14T01:31:43.348 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48716.log 2023-10-14T01:31:43.348 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56260.log.gz 2023-10-14T01:31:43.349 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56072.log.gz 2023-10-14T01:31:43.349 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48573.log 2023-10-14T01:31:43.349 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49394.log 2023-10-14T01:31:43.349 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48716.log.gz 2023-10-14T01:31:43.350 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44271.log 2023-10-14T01:31:43.350 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48573.log.gz 2023-10-14T01:31:43.350 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34413.log 2023-10-14T01:31:43.351 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49394.log.gz 2023-10-14T01:31:43.351 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39993.log 2023-10-14T01:31:43.351 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44271.log.gz 2023-10-14T01:31:43.351 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.34413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55884.log 2023-10-14T01:31:43.352 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34413.log.gz 2023-10-14T01:31:43.352 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50615.log 2023-10-14T01:31:43.352 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39993.log.gz 2023-10-14T01:31:43.352 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47121.log 2023-10-14T01:31:43.353 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55884.log.gz 2023-10-14T01:31:43.353 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50615.log.gz 2023-10-14T01:31:43.353 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49149.log 2023-10-14T01:31:43.353 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.47121.log: /var/log/ceph/ceph-client.admin.38773.log 2023-10-14T01:31:43.354 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47121.log.gz 2023-10-14T01:31:43.354 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50795.log 2023-10-14T01:31:43.354 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49149.log.gz 2023-10-14T01:31:43.355 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39151.log 2023-10-14T01:31:43.355 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38773.log.gz 2023-10-14T01:31:43.355 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38790.log 2023-10-14T01:31:43.355 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50795.log.gz 2023-10-14T01:31:43.356 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48052.log 2023-10-14T01:31:43.356 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39151.log.gz 2023-10-14T01:31:43.356 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37481.log 2023-10-14T01:31:43.356 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38790.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38790.log.gz 2023-10-14T01:31:43.357 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54580.log 2023-10-14T01:31:43.357 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48052.log.gz 2023-10-14T01:31:43.357 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39653.log 2023-10-14T01:31:43.357 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37481.log.gz 2023-10-14T01:31:43.358 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.54580.log: -- /var/log/ceph/ceph-client.admin.46523.log 2023-10-14T01:31:43.358 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54580.log.gz 2023-10-14T01:31:43.358 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55166.log 2023-10-14T01:31:43.359 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39653.log.gz 2023-10-14T01:31:43.359 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46523.log.gz 2023-10-14T01:31:43.359 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51617.log 2023-10-14T01:31:43.360 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56418.log 2023-10-14T01:31:43.360 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55166.log.gz 2023-10-14T01:31:43.360 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53363.log 2023-10-14T01:31:43.360 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51617.log.gz 2023-10-14T01:31:43.361 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45477.log 2023-10-14T01:31:43.361 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56418.log.gz 2023-10-14T01:31:43.361 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52466.log 2023-10-14T01:31:43.362 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42094.log 2023-10-14T01:31:43.362 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45477.log.gz 2023-10-14T01:31:43.362 INFO:teuthology.orchestra.run.smithi138.stderr: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.53363.log.gz 2023-10-14T01:31:43.362 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47642.log 2023-10-14T01:31:43.363 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52466.log.gz 2023-10-14T01:31:43.363 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40045.log 2023-10-14T01:31:43.363 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42094.log.gz 2023-10-14T01:31:43.364 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2023-10-14T01:31:43.364 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47642.log.gz 2023-10-14T01:31:43.364 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49918.log 2023-10-14T01:31:43.364 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40045.log.gz 2023-10-14T01:31:43.365 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43032.log 2023-10-14T01:31:43.365 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2023-10-14T01:31:43.365 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34508.log 2023-10-14T01:31:43.366 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49918.log.gz 2023-10-14T01:31:43.366 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48777.log 2023-10-14T01:31:43.366 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43032.log.gz 2023-10-14T01:31:43.366 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.34508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34508.log.gz 2023-10-14T01:31:43.367 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49586.log 2023-10-14T01:31:43.367 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53020.log 2023-10-14T01:31:43.367 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48777.log.gz 2023-10-14T01:31:43.368 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54285.log 2023-10-14T01:31:43.368 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49586.log.gz 2023-10-14T01:31:43.368 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53020.log.gz 2023-10-14T01:31:43.368 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48606.log 2023-10-14T01:31:43.369 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38898.log 2023-10-14T01:31:43.369 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54285.log.gz 2023-10-14T01:31:43.369 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46575.log 2023-10-14T01:31:43.370 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48606.log.gz 2023-10-14T01:31:43.370 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38898.log.gz 2023-10-14T01:31:43.370 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51163.log 2023-10-14T01:31:43.370 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38844.log 2023-10-14T01:31:43.371 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46575.log.gz 2023-10-14T01:31:43.371 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51010.log 2023-10-14T01:31:43.371 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51163.log.gz 2023-10-14T01:31:43.371 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38844.log.gz 2023-10-14T01:31:43.372 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47889.log 2023-10-14T01:31:43.372 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2023-10-14T01:31:43.372 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51010.log.gz 2023-10-14T01:31:43.373 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53337.log 2023-10-14T01:31:43.373 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47889.log.gz 2023-10-14T01:31:43.373 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37670.log 2023-10-14T01:31:43.373 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2023-10-14T01:31:43.374 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45933.log 2023-10-14T01:31:43.374 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53337.log.gz 2023-10-14T01:31:43.374 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53790.log 2023-10-14T01:31:43.375 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37670.log.gz 2023-10-14T01:31:43.375 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55032.log 2023-10-14T01:31:43.375 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45933.log.gz 2023-10-14T01:31:43.375 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2023-10-14T01:31:43.376 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53790.log.gz 2023-10-14T01:31:43.376 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.55032.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37845.log 2023-10-14T01:31:43.376 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55032.log.gz 2023-10-14T01:31:43.376 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 2023-10-14T01:31:43.377 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2023-10-14T01:31:43.377 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37845.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37845.log.gz 2023-10-14T01:31:43.377 INFO:teuthology.orchestra.run.smithi138.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43002.log 2023-10-14T01:31:43.378 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37507.log 2023-10-14T01:31:43.378 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2023-10-14T01:31:43.378 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45720.log 2023-10-14T01:31:43.378 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43002.log.gz 2023-10-14T01:31:43.379 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55219.log 2023-10-14T01:31:43.379 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37507.log.gz 2023-10-14T01:31:43.379 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41844.log 2023-10-14T01:31:43.379 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45720.log.gz 2023-10-14T01:31:43.380 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55219.log.gz 2023-10-14T01:31:43.380 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43962.log 2023-10-14T01:31:43.380 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41844.log: /var/log/ceph/ceph-client.admin.42206.log 2023-10-14T01:31:43.381 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41844.log.gz 2023-10-14T01:31:43.381 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45558.log 2023-10-14T01:31:43.381 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43962.log.gz 2023-10-14T01:31:43.381 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43188.log 2023-10-14T01:31:43.382 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42206.log.gz 2023-10-14T01:31:43.382 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46244.log 2023-10-14T01:31:43.382 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45558.log.gz 2023-10-14T01:31:43.382 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43493.log 2023-10-14T01:31:43.383 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43188.log.gz 2023-10-14T01:31:43.383 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45832.log 2023-10-14T01:31:43.383 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46244.log.gz 2023-10-14T01:31:43.384 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.43493.log: -- /var/log/ceph/ceph-client.admin.47211.log 2023-10-14T01:31:43.384 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43493.log.gz 2023-10-14T01:31:43.384 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46355.log 2023-10-14T01:31:43.384 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45832.log.gz 2023-10-14T01:31:43.385 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48299.log 2023-10-14T01:31:43.385 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47211.log.gz 2023-10-14T01:31:43.385 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.46355.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51253.log 2023-10-14T01:31:43.385 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46355.log.gz 2023-10-14T01:31:43.386 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52968.log 2023-10-14T01:31:43.386 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48299.log.gz 2023-10-14T01:31:43.386 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38483.log 2023-10-14T01:31:43.386 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51253.log.gz 2023-10-14T01:31:43.387 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41820.log 2023-10-14T01:31:43.387 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52968.log.gz 2023-10-14T01:31:43.387 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55245.log 2023-10-14T01:31:43.388 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38483.log.gz 2023-10-14T01:31:43.388 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50340.log 2023-10-14T01:31:43.388 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41820.log.gz 2023-10-14T01:31:43.388 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45373.log 2023-10-14T01:31:43.389 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55245.log.gz 2023-10-14T01:31:43.389 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54448.log 2023-10-14T01:31:43.389 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50340.log.gz 2023-10-14T01:31:43.389 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50155.log 2023-10-14T01:31:43.390 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45373.log.gz 2023-10-14T01:31:43.390 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54448.log.gz 2023-10-14T01:31:43.390 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56232.log 2023-10-14T01:31:43.390 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50155.log: -- /var/log/ceph/ceph-client.admin.47560.log 2023-10-14T01:31:43.391 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50155.log.gz 2023-10-14T01:31:43.391 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44977.log 2023-10-14T01:31:43.391 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56232.log.gz 2023-10-14T01:31:43.391 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37592.log 2023-10-14T01:31:43.392 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47560.log.gz 2023-10-14T01:31:43.392 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44977.log: /var/log/ceph/ceph-client.admin.37671.log 2023-10-14T01:31:43.392 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44977.log.gz 2023-10-14T01:31:43.392 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2023-10-14T01:31:43.393 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37592.log.gz 2023-10-14T01:31:43.393 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37671.log.gz 2023-10-14T01:31:43.393 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54607.log 2023-10-14T01:31:43.393 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53947.log 2023-10-14T01:31:43.394 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2023-10-14T01:31:43.394 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38628.log 2023-10-14T01:31:43.394 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54607.log.gz 2023-10-14T01:31:43.395 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2023-10-14T01:31:43.395 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53947.log.gz 2023-10-14T01:31:43.395 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49481.log 2023-10-14T01:31:43.395 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38628.log.gz 2023-10-14T01:31:43.395 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37749.log 2023-10-14T01:31:43.396 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2023-10-14T01:31:43.396 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50705.log 2023-10-14T01:31:43.396 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49481.log.gz 2023-10-14T01:31:43.397 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50765.log 2023-10-14T01:31:43.397 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.37749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37749.log.gz 2023-10-14T01:31:43.397 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38537.log 2023-10-14T01:31:43.397 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50705.log.gz 2023-10-14T01:31:43.398 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55940.log 2023-10-14T01:31:43.398 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50765.log.gz 2023-10-14T01:31:43.398 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47612.log 2023-10-14T01:31:43.398 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38537.log.gz 2023-10-14T01:31:43.399 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55831.log 2023-10-14T01:31:43.399 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55940.log.gz 2023-10-14T01:31:43.399 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52687.log 2023-10-14T01:31:43.399 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47612.log.gz 2023-10-14T01:31:43.400 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54419.log 2023-10-14T01:31:43.400 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55831.log.gz 2023-10-14T01:31:43.400 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47364.log 2023-10-14T01:31:43.401 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52687.log.gz 2023-10-14T01:31:43.401 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2023-10-14T01:31:43.401 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54419.log.gz 2023-10-14T01:31:43.401 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47364.log.gz 2023-10-14T01:31:43.402 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38969.log 2023-10-14T01:31:43.402 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45444.log 2023-10-14T01:31:43.402 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2023-10-14T01:31:43.403 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46878.log 2023-10-14T01:31:43.403 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38969.log.gz 2023-10-14T01:31:43.403 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53415.log 2023-10-14T01:31:43.403 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45444.log.gz 2023-10-14T01:31:43.403 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52286.log 2023-10-14T01:31:43.404 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46878.log.gz 2023-10-14T01:31:43.404 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53415.log.gz 2023-10-14T01:31:43.404 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44893.log 2023-10-14T01:31:43.404 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42363.log 2023-10-14T01:31:43.405 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52286.log.gz 2023-10-14T01:31:43.405 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52651.log 2023-10-14T01:31:43.405 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44893.log.gz 2023-10-14T01:31:43.405 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42363.log.gz 2023-10-14T01:31:43.406 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46079.log 2023-10-14T01:31:43.406 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52577.log 2023-10-14T01:31:43.406 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52651.log.gz 2023-10-14T01:31:43.407 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46079.log.gz 2023-10-14T01:31:43.407 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56205.log 2023-10-14T01:31:43.407 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53178.log 2023-10-14T01:31:43.407 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52577.log.gz 2023-10-14T01:31:43.408 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42307.log 2023-10-14T01:31:43.408 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56205.log.gz 2023-10-14T01:31:43.408 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38409.log 2023-10-14T01:31:43.408 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53178.log.gz 2023-10-14T01:31:43.409 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49757.log 2023-10-14T01:31:43.409 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42307.log.gz 2023-10-14T01:31:43.409 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2023-10-14T01:31:43.409 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38409.log.gz 2023-10-14T01:31:43.410 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45989.log 2023-10-14T01:31:43.410 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49757.log.gz 2023-10-14T01:31:43.410 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40778.log 2023-10-14T01:31:43.410 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2023-10-14T01:31:43.411 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56126.log 2023-10-14T01:31:43.411 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45989.log.gz 2023-10-14T01:31:43.413 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40778.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.47980.log 2023-10-14T01:31:43.414 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40778.log.gz 2023-10-14T01:31:43.414 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56126.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56126.log.gz -5 2023-10-14T01:31:43.414 INFO:teuthology.orchestra.run.smithi138.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38935.log 2023-10-14T01:31:43.415 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47980.log.gz 2023-10-14T01:31:43.415 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41965.log 2023-10-14T01:31:43.415 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45961.log 2023-10-14T01:31:43.416 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38935.log.gz 2023-10-14T01:31:43.416 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39347.log 2023-10-14T01:31:43.416 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41965.log.gz 2023-10-14T01:31:43.416 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50246.log 2023-10-14T01:31:43.417 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45961.log.gz 2023-10-14T01:31:43.417 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43092.log 2023-10-14T01:31:43.417 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39347.log.gz 2023-10-14T01:31:43.417 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52135.log 2023-10-14T01:31:43.418 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50246.log.gz 2023-10-14T01:31:43.418 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50490.log 2023-10-14T01:31:43.418 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43092.log.gz 2023-10-14T01:31:43.418 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47919.log 2023-10-14T01:31:43.419 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52135.log.gz 2023-10-14T01:31:43.419 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48881.log 2023-10-14T01:31:43.419 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50490.log.gz 2023-10-14T01:31:43.419 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47830.log 2023-10-14T01:31:43.420 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47919.log.gz 2023-10-14T01:31:43.420 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.48881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47307.log 2023-10-14T01:31:43.420 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48881.log.gz 2023-10-14T01:31:43.420 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45528.log 2023-10-14T01:31:43.421 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47830.log.gz 2023-10-14T01:31:43.421 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50886.log 2023-10-14T01:31:43.421 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47307.log.gz 2023-10-14T01:31:43.422 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41356.log 2023-10-14T01:31:43.422 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45528.log.gz 2023-10-14T01:31:43.422 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52166.log 2023-10-14T01:31:43.422 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50886.log.gz 2023-10-14T01:31:43.423 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2023-10-14T01:31:43.423 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41356.log.gz 2023-10-14T01:31:43.423 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39006.log 2023-10-14T01:31:43.424 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52166.log.gz 2023-10-14T01:31:43.424 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47395.log 2023-10-14T01:31:43.424 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2023-10-14T01:31:43.424 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52347.log 2023-10-14T01:31:43.425 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39006.log.gz 2023-10-14T01:31:43.425 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41166.log 2023-10-14T01:31:43.425 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47395.log.gz 2023-10-14T01:31:43.425 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43808.log 2023-10-14T01:31:43.426 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52347.log.gz 2023-10-14T01:31:43.426 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45312.log 2023-10-14T01:31:43.426 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41166.log.gz 2023-10-14T01:31:43.426 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46019.log 2023-10-14T01:31:43.427 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43808.log.gz 2023-10-14T01:31:43.427 INFO:teuthology.orchestra.run.smithi138.stderr:gzip/var/log/ceph/ceph-client.admin.45312.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47040.log 2023-10-14T01:31:43.427 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45312.log.gz 2023-10-14T01:31:43.427 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55087.log 2023-10-14T01:31:43.428 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46019.log.gz 2023-10-14T01:31:43.428 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47040.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47040.log.gz 2023-10-14T01:31:43.428 INFO:teuthology.orchestra.run.smithi138.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49666.log 2023-10-14T01:31:43.428 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40504.log 2023-10-14T01:31:43.429 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55087.log.gz 2023-10-14T01:31:43.429 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42667.log 2023-10-14T01:31:43.429 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49666.log.gz 2023-10-14T01:31:43.429 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.40504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46663.log 2023-10-14T01:31:43.430 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40504.log.gz 2023-10-14T01:31:43.430 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44022.log 2023-10-14T01:31:43.430 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42667.log.gz 2023-10-14T01:31:43.431 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55297.log 2023-10-14T01:31:43.431 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46663.log.gz 2023-10-14T01:31:43.431 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46768.log 2023-10-14T01:31:43.432 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44022.log.gz 2023-10-14T01:31:43.432 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41259.log 2023-10-14T01:31:43.432 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55297.log.gz 2023-10-14T01:31:43.432 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.46768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44949.log 2023-10-14T01:31:43.433 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46768.log.gz 2023-10-14T01:31:43.433 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45403.log 2023-10-14T01:31:43.433 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41259.log.gz 2023-10-14T01:31:43.433 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47012.log 2023-10-14T01:31:43.434 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44949.log.gz 2023-10-14T01:31:43.434 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53686.log 2023-10-14T01:31:43.434 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45403.log.gz 2023-10-14T01:31:43.435 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43992.log 2023-10-14T01:31:43.435 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47012.log.gz 2023-10-14T01:31:43.435 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54815.log 0.0% 2023-10-14T01:31:43.435 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53686.log.gz 2023-10-14T01:31:43.436 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42893.log 2023-10-14T01:31:43.436 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43992.log.gz 2023-10-14T01:31:43.436 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47778.log 2023-10-14T01:31:43.437 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54815.log.gz 2023-10-14T01:31:43.437 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49942.log 2023-10-14T01:31:43.437 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42893.log.gz 2023-10-14T01:31:43.437 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.47778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42637.log 2023-10-14T01:31:43.438 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47778.log.gz 2023-10-14T01:31:43.438 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.49942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42554.log 2023-10-14T01:31:43.438 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49942.log.gz 2023-10-14T01:31:43.438 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42637.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50460.log 2023-10-14T01:31:43.439 INFO:teuthology.orchestra.run.smithi138.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42637.log.gz 2023-10-14T01:31:43.439 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.42554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45771.log 2023-10-14T01:31:43.439 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42554.log.gz 2023-10-14T01:31:43.440 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.50460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51133.log 2023-10-14T01:31:43.440 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50460.log.gz 2023-10-14T01:31:43.440 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.45771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51798.log 2023-10-14T01:31:43.440 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45771.log.gz 2023-10-14T01:31:43.441 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44921.log 2023-10-14T01:31:43.441 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51133.log.gz 2023-10-14T01:31:43.441 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43247.log 2023-10-14T01:31:43.442 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51798.log.gz 2023-10-14T01:31:43.442 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39259.log 2023-10-14T01:31:43.442 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44921.log.gz 2023-10-14T01:31:43.443 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54924.log 2023-10-14T01:31:43.443 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43247.log.gz 2023-10-14T01:31:43.443 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54763.log 2023-10-14T01:31:43.443 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39259.log.gz 2023-10-14T01:31:43.444 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51982.log 2023-10-14T01:31:43.444 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54924.log.gz 2023-10-14T01:31:43.444 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55913.log 2023-10-14T01:31:43.444 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54763.log.gz 2023-10-14T01:31:43.444 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.51982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52994.log 2023-10-14T01:31:43.445 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51982.log.gz 2023-10-14T01:31:43.445 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.55913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54339.log 2023-10-14T01:31:43.445 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55913.log.gz 2023-10-14T01:31:43.446 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41718.log 2023-10-14T01:31:43.446 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52994.log.gz 2023-10-14T01:31:43.446 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.54339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44519.log 2023-10-14T01:31:43.446 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54339.log.gz 2023-10-14T01:31:43.447 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39401.log 2023-10-14T01:31:43.447 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.41718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41718.log.gz 2023-10-14T01:31:43.447 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.44519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52484.log 2023-10-14T01:31:43.447 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44519.log.gz 2023-10-14T01:31:43.448 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.39401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43934.log 2023-10-14T01:31:43.448 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39401.log.gz 2023-10-14T01:31:43.448 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.52484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53603.log 2023-10-14T01:31:43.448 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52484.log.gz 2023-10-14T01:31:43.448 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.43934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43934.log.gz 2023-10-14T01:31:43.449 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56044.log 2023-10-14T01:31:43.449 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53603.log.gz 2023-10-14T01:31:43.449 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38719.log 2023-10-14T01:31:43.449 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.56044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53920.log 2023-10-14T01:31:43.450 INFO:teuthology.orchestra.run.smithi138.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56044.log.gz 2023-10-14T01:31:43.450 INFO:teuthology.orchestra.run.smithi138.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34591.log 2023-10-14T01:31:43.450 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.38719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38719.log.gz 2023-10-14T01:31:43.450 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.53920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53920.log.gz 2023-10-14T01:31:43.451 INFO:teuthology.orchestra.run.smithi138.stderr:/var/log/ceph/ceph-client.admin.34591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34591.log.gz 2023-10-14T01:31:43.451 INFO:teuthology.orchestra.run.smithi138.stderr: 90.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-14T01:32:03.851 INFO:teuthology.orchestra.run.smithi138.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-14T01:32:10.354 INFO:teuthology.orchestra.run.smithi138.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-14T02:16:09.040 INFO:teuthology.orchestra.run.smithi138.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-14T02:19:23.894 INFO:teuthology.orchestra.run.smithi138.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-14T02:27:06.171 INFO:teuthology.orchestra.run.smithi138.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-14T02:27:06.192 INFO:teuthology.orchestra.run.smithi138.stderr: 2023-10-14T02:27:06.192 INFO:teuthology.orchestra.run.smithi138.stderr:real 55m23.309s 2023-10-14T02:27:06.193 INFO:teuthology.orchestra.run.smithi138.stderr:user 44m14.619s 2023-10-14T02:27:06.193 INFO:teuthology.orchestra.run.smithi138.stderr:sys 2m15.118s 2023-10-14T02:27:06.193 INFO:tasks.ceph:Archiving logs... 2023-10-14T02:27:06.195 DEBUG:teuthology.misc:Transferring archived files from smithi138:/var/log/ceph to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257/remote/smithi138/log 2023-10-14T02:27:06.198 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-14T02:29:36.227 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T02:29:36.247 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-14T02:29:36.247 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-14T02:29:36.324 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-14T02:29:36.334 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-14T02:29:36.335 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T02:29:36.398 INFO:teuthology.orchestra.run.smithi138.stdout: remote refid st t when poll reach delay offset jitter 2023-10-14T02:29:36.399 INFO:teuthology.orchestra.run.smithi138.stdout:============================================================================== 2023-10-14T02:29:36.399 INFO:teuthology.orchestra.run.smithi138.stdout:*hv01.front.sepi 44.4.53.2 2 u 1 128 377 0.182 -4.609 1.193 2023-10-14T02:29:36.399 INFO:teuthology.orchestra.run.smithi138.stdout:+hv02.front.sepi 63.231.80.2 3 u 209 256 377 0.198 -5.374 0.848 2023-10-14T02:29:36.400 INFO:teuthology.orchestra.run.smithi138.stdout:+hv03.front.sepi 74.6.168.72 3 u 135 256 377 0.178 -5.441 0.616 2023-10-14T02:29:36.400 INFO:teuthology.orchestra.run.smithi138.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-14T02:29:36.400 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-14T02:29:36.410 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-14T02:29:36.410 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-14T02:29:36.419 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-14T02:29:36.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-14T02:29:36.438 INFO:teuthology.task.internal:Duration was 9801.631852 seconds 2023-10-14T02:29:36.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-14T02:29:36.448 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-14T02:29:36.448 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-14T02:29:36.778 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-14T02:29:36.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2023-10-14T02:29:36.779 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-14T02:29:36.851 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-14T02:29:36.851 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-14T02:29:36.996 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-14T02:29:36.997 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-14T02:29:37.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-14T02:29:37.628 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-14T02:29:37.629 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-14T02:29:37.686 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-14T02:29:37.723 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-14T02:29:37.741 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2023-10-14T02:29:37.780 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-14T02:29:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T02:29:37.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-14T02:29:37.842 INFO:teuthology.task.internal:Transferring archived files... 2023-10-14T02:29:37.844 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257/remote/smithi138 2023-10-14T02:29:37.844 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-14T02:29:37.895 INFO:teuthology.task.internal:Removing archive directory... 2023-10-14T02:29:37.896 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-14T02:29:37.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-14T02:29:37.954 INFO:teuthology.task.internal:Not uploading archives. 2023-10-14T02:29:37.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-14T02:29:37.964 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-14T02:29:37.965 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-14T02:29:38.006 INFO:teuthology.orchestra.run.smithi138.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 14 02:29 /home/ubuntu/cephtest 2023-10-14T02:29:38.006 INFO:teuthology.orchestra.run.smithi138.stdout: 673331 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 13 23:56 /home/ubuntu/cephtest/mnt.0 2023-10-14T02:29:38.007 INFO:teuthology.orchestra.run.smithi138.stdout: 673755 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 13 23:57 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-14T02:29:38.007 INFO:teuthology.orchestra.run.smithi138.stdout: 687661 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 13 23:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-14T02:29:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T02:29:38.008 INFO:teuthology.orchestra.run.smithi138.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-14T02:29:38.009 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi138 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-14T02:29:38.017 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-14T02:29:38.028 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-14T02:29:38.039 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi138 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-14T02:29:38.039 INFO:teuthology.nuke:Checking targets against current locks 2023-10-14T02:29:38.071 DEBUG:teuthology.nuke:shortname: smithi138 2023-10-14T02:29:38.072 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T02:29:38.094 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427257', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-13 23:37:44.811141', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIbphirixlHwnOwOOVhbCQiR2axA3wmHarKaDteDBvLWHZeDd90LRhhgzWrE6nKf214ifB1/ugwsfdEyiLumX0='} 2023-10-14T02:29:38.149 INFO:teuthology.orchestra.console.smithi138:Power off 2023-10-14T02:29:38.151 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-14T02:29:38.175 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off 2023-10-14T02:29:38.176 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:29:38.202 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2023-10-14T02:29:42.204 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:29:42.315 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2023-10-14T02:29:46.320 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:29:46.431 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2023-10-14T02:29:50.435 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:29:50.545 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2023-10-14T02:29:50.646 INFO:teuthology.orchestra.console.smithi138:Power off completed 2023-10-14T02:29:50.752 INFO:teuthology.run:Summary data: description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/none features/journaling msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 9801.631851673126 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi138 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=df6af9e7556644af8d34e472e8766dbf status: fail success: false 2023-10-14T02:29:50.753 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-14T02:29:50.826 INFO:teuthology.run:FAIL